Oct 31 00:28:58.404670 ip-10-0-131-97 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:28:58.882119 ip-10-0-131-97 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:58.882119 ip-10-0-131-97 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:28:58.882119 ip-10-0-131-97 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:58.882119 ip-10-0-131-97 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:28:58.886813 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886675    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.886813 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886813    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.886813 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886816    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886820    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886823    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886826    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886828    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886831    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886833    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886836    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886839    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886841    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886844    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886846    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886849    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886852    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886855    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886857    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886859    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886862    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886868    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.886901 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886871    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886874    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886876    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886879    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886881    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886883    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886888    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886891    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886894    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886897    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886899    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886902    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886905    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886907    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886910    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886913    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886915    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886918    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886920    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.887322 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886923    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886925    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886927    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886930    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886932    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886935    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886937    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886939    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886942    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886945    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886947    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886950    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886953    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886956    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886959    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886962    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886964    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886966    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886969    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886971    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.887765 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886975    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886977    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886980    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886982    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886984    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886987    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886989    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886991    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886994    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886996    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.886998    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887001    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887003    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887006    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887008    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887010    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887013    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887015    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887017    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887020    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.888249 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887022    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887025    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887029    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887031    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887034    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887037    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887040    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887043    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887045    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887047    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887050    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887053    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887056    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887059    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887061    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887063    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887066    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887068    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887071    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.888757 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887075    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887079    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887082    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887084    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887087    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887089    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.887093    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888008    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888016    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888020    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888022    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888025    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888028    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888030    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888033    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888035    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888038    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888040    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888043    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.889190 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888049    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888052    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888055    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888057    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888060    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888062    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888065    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888068    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888071    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888074    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888076    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888079    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888081    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888084    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888086    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888088    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888091    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888094    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888096    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.889664 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888098    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888100    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888103    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888105    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888108    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888110    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888112    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888115    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888117    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888120    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888122    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888124    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888127    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888129    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888137    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888140    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888142    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888145    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888147    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.890109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888150    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888152    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888155    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888157    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888160    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888162    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888164    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888167    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888170    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888174    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888177    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888180    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888184    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888188    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888191    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888194    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888196    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888199    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888201    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.890585 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888204    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888206    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888209    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888211    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888214    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888216    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888219    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888221    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888224    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888227    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888229    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888232    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888234    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888239    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888242    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888244    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888247    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888249    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888252    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.891062 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888255    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888257    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888260    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888263    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888266    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888268    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888270    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888273    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888276    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888278    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888281    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888283    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888285    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888288    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888290    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888292    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888312    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888316    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888871    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888878    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.891506 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888881    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888884    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888888    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888891    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888894    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888897    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888902    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888905    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888908    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888912    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888915    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888917    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888920    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888922    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888925    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888927    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888929    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888932    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888934    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.891967 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888937    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888939    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888942    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888944    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888946    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888949    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888951    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888953    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888956    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888958    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888960    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888963    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888965    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888967    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888972    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888975    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888978    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888980    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888982    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888985    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.892421 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888988    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888990    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888993    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888995    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.888997    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889000    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889002    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889004    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889007    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889009    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889012    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889014    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889017    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889019    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889021    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889024    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889026    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889028    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889030    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889033    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.892893 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889036    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889038    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889040    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889043    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889045    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889047    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889052    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889054    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889057    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889060    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889063    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889065    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889068    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889070    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889073    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889075    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889078    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889080    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889082    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889085    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.893380 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889087    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889089    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889092    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889094    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889097    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889099    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889101    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889105    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889108    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889111    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889114    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889116    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889119    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889121    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889123    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889127    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889129    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889131    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889135    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.893839 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889137    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889140    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889142    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889145    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889147    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.889150    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890129    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890141    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890147    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890152    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890157    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890160    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890164    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890168    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890171    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890174    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890178    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890181    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890184    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890187    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890189    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890192    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890195    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:28:58.894280 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890198    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890203    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890206    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890209    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890211    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890216    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890221    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890224    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890228    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890231    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890234    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890237    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890240    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890244    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890247    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890251    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890254    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890256    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890259    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890262    2576 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890265    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890269    2576 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890272    2576 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890275    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:28:58.894830 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890279    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890281    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890286    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890289    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890292    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890295    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890298    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890301    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890304    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890307    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890310    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890313    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890315    2576 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890320    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890323    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890326    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890330    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890333    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890336    2576 flags.go:64] FLAG: --help="false"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890339    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890342    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890356    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890361    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:28:58.895459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890364    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890367    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890370    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890373    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890376    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890379    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890381    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890385    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890387    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890390    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890393    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890396    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890398    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890401    2576 flags.go:64] FLAG: --lock-file=""
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890404    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890406    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890410    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890416    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890418    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890421    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890424    2576 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890427    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890431    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890434    2576 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890437    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:28:58.895984 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890441    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890444    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890448    2576 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890450    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890454    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890456    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890459    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890465    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890468    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890471    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890478    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890481    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890484    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890487    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890490    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890493    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890496    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890498    2576 flags.go:64] FLAG: --port="10250"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890501    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890504    2576 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0daf3ae3370a4ac65"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890507    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890509    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890512    2576 flags.go:64] FLAG: --register-node="true"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890515    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890519    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890522    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:28:58.896625 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890525    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890528    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890531    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890535    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890538    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890541    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890544    2576 flags.go:64] FLAG: --runonce="false"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890547    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890550    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890553    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890556    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890559    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890562    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890565    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890568    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890571    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890574    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890577    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890580    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890582    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890585    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890589    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890594    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890597    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890600    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:28:58.897231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890604    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890607    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890609    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890612    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890615    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890618    2576 flags.go:64] FLAG: --v="2"
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890622    2576 flags.go:64] FLAG: --version="false"
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890626    2576 flags.go:64] FLAG: --vmodule=""
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890630    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.890633    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890773    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890776    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890779    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890785    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890787    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890790    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890792    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890795    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890798    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890802    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890805    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.897831 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890809    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890812    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890815    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890818    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890820    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890823    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890825    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890828    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890830    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890832    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890835    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890837    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890840    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890842    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890845    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890847    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890850    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890852    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890854    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890857    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.898447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890859    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890863    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890866    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890868    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890872    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890875    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890877    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890879    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890882    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890884    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890886    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890889    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890891    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890894    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890897    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890899    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890902    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890904    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890907    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890909    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.898920 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890911    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890915    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890918    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890921    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890924    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890926    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890929    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890931    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890934    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890936    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890939    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890941    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890944    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890948    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890950    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890952    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890956    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890959    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890961    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890964    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.899389 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890966    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890968    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890971    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890973    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890976    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890978    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890981    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890983    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890986    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890988    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890990    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890993    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890995    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.890997    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891000    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891002    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891005    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891007    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891010    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891012    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.899895 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891014    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891017    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891019    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891022    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891024    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891027    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891030    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891032    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891036    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891038    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891041    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891043    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891046    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891048    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891051    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.900500 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.891076    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891112    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891116    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891119    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891121    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891125    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891128    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891130    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891132    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891135    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891138    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891140    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891143    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891145    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891148    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891150    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891152    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891155    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891157    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.900849 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891160    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891163    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891167    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891172    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891175    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891178    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891180    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891186    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891188    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891191    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891193    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891196    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891198    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891201    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891203    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891205    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891208    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891210    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891212    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.901282 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891215    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891218    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891220    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891222    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891225    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891228    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891230    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891233    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891235    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891238    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891240    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891243    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891245    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891248    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891250    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891252    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891256    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891258    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891261    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891263    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.901724 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891267    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891270    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891273    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891275    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891277    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891281    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891284    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891287    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891290    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891292    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891295    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891297    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891299    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891302    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891305    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891308    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891310    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891313    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891315    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.902204 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891318    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891320    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891322    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891325    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891327    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891330    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891332    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891334    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891337    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891340    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891343    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891357    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891360    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891363    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891366    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891369    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891371    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891373    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891376    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891378    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.902648 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891381    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891383    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891386    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891388    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891390    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891393    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891396    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891400    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891403    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.891421    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.897938    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.897956    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.898562    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.898568    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898627    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898632    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898636    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898639    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898642    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.903121 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898646    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898648    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898651    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898654    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898656    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898659    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898661    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898663    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898667    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898669    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898672    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898674    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898677    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898679    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898681    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898684    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898686    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898689    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898691    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898694    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.903569 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898697    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898699    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898702    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898704    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898707    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898710    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898712    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898715    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898717    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898720    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898723    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898726    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898728    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898730    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898733    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898735    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898737    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898740    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898744    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.904034 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898748    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898751    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898754    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898757    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898759    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898762    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898764    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898767    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898769    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898773    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898776    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898780    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898783    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898785    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898788    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898791    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898793    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898796    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898799    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.904570 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898802    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898804    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898806    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898809    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898811    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898814    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898817    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898819    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898821    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898824    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898826    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898829    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898831    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898834    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898836    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898839    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898842    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898844    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898847    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.905052 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898849    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898852    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898854    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898857    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898859    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898862    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898864    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898867    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898869    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898872    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898875    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898878    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898880    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898883    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898885    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898888    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898890    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898893    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898895    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.905809 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898897    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898900    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898902    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898904    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.898907    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.898933    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899088    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899093    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899096    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899100    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899103    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899106    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899108    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899112    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899114    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.906499 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899116    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899119    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899122    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899124    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899127    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899129    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899131    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899134    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899139    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899142    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899144    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899147    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899149    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899151    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899154    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899156    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899158    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899160    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899163    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.907043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899165    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899167    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899170    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899173    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899175    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899177    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899180    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899183    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899186    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899188    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899190    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899193    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899195    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899198    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899200    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899202    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899205    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899207    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899210    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899212    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.907599 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899215    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899217    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899220    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899223    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899225    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899228    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899230    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899232    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899235    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899237    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899239    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899242    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899244    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899246    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899249    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899251    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899253    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899256    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899258    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899261    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899264    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.908109 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899266    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899268    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899271    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899274    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899276    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899278    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899281    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899283    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899286    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899288    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899290    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899293    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899295    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899299    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899302    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899305    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899309    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899312    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899315    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.909208 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899318    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899321    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899323    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899326    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899328    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899331    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899333    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899335    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899338    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899340    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899342    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899360    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899365    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899369    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899372    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899375    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899377    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.909653 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899380    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.899403    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899437    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899440    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899443    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899445    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899448    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899451    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899454    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899456    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899459    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899461    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899464    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899466    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899469    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899471    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:28:58.910043 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899474    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899476    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899478    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899481    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899483    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899485    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899488    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899490    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899493    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899495    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899498    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899500    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899502    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899505    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899507    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899510    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899513    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899515    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899517    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899520    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899522    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:28:58.910439 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899524    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899527    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899529    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899531    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899534    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899536    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899539    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899541    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899544    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899546    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899548    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899551    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899553    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899556    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899560    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899563    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899567    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899570    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899573    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:28:58.910907 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899575    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899578    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899581    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899584    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899586    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899588    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899590    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899593    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899595    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899598    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899600    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899603    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899605    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899607    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899610    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899612    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899614    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899617    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899619    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:28:58.911447 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899622    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899624    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899627    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899629    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899632    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899634    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899637    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899639    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899641    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899644    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899646    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899648    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899650    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899653    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899656    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899658    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899661    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899663    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899665    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:28:58.911913 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899668    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899670    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899673    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899675    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899677    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899679    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899682    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899684    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899687    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899689    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899691    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899694    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899696    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:28:58.899698    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.899752    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.901752    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.902644    2576 server.go:1014] "Starting client certificate rotation"
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.902744    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:58.912371 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.903938    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:28:58.912784 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.906370    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:58.912784 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.908715    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:28:58.923961 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.923938    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:28:58.927827 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.927803    2576 log.go:25] "Validated CRI v1 image API"
Oct 31 00:28:58.928726 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.928701    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:28:58.931815 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.931794    2576 fs.go:135] Filesystem UUIDs: map[41126c8c-219f-4105-8c86-6be80465a1a4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d53334c-f11a-4089-9695-4bdf1e2b2636:/dev/nvme0n1p4]
Oct 31 00:28:58.931878 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.931814    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 00:28:58.937783 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.937663    2576 manager.go:217] Machine: {Timestamp:2026-10-31 00:28:58.935379819 +0000 UTC m=+0.382762749 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199764 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b89156c6c6480c8ff508e9a3ce798 SystemUUID:ec2b8915-6c6c-6480-c8ff-508e9a3ce798 BootID:00c39938-a3ea-4d24-b3bc-d4123b5391ac Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cf:f3:51:0d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cf:f3:51:0d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:b4:51:04:88:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:28:58.937783 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.937781    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:28:58.937894 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.937868    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:28:58.940858 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.940825    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:28:58.941039 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.940859    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-97.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:28:58.941121 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.941044    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:28:58.941121 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.941056    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:28:58.941121 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.941101    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:58.941225 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.941127    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:28:58.942144 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.942132    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:28:58.946090 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.946076    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:28:58.946228 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.946217    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:28:58.947766 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.947755    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:28:58.947828 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.947774    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:28:58.947828 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.947789    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:28:58.947828 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.947803    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:28:58.947828 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.947815    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:28:58.948194 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.948181    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:58.951458 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.951343    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:28:58.956895 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.956878    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:28:58.957706 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957694    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:28:58.957743 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957724    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:28:58.957887 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957871    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:28:58.957887 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957888    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:28:58.957887 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957894    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:28:58.958032 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957900    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:28:58.958032 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957907    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:28:58.958032 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.957912    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:28:58.958479 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.958470    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:28:58.958509 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.958479    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:28:58.958509 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.958487    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:28:58.958509 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.958493    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:28:58.958509 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.958500    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:28:58.958509 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.958508    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:28:58.959070 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.959062    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:28:58.959070 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.959071    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:28:58.962852 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.962838    2576 server.go:1263] "Started kubelet"
Oct 31 00:28:58.962970 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.962928    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:28:58.962970 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.962937    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:28:58.963067 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.963011    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:28:58.963556 ip-10-0-131-97 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:28:58.964589 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.964408    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:28:58.964925 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.964911    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:28:58.970303 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.970284    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:28:58.970743 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.970722    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:28:58.970841 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:58.970784    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 00:28:58.970841 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:58.970826    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found"
Oct 31 00:28:58.971430 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971414    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:28:58.971430 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971417    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:28:58.971567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971443    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:28:58.971567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971513    2576 factory.go:55] Registering systemd factory
Oct 31 00:28:58.971567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971541    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:28:58.971567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971552    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:28:58.971567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971552    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:28:58.971757 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971750    2576 factory.go:153] Registering CRI-O factory
Oct 31 00:28:58.971801 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971763    2576 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:28:58.971844 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971812    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:28:58.971844 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971831    2576 factory.go:103] Registering Raw factory
Oct 31 00:28:58.971923 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.971846    2576 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:28:58.972768 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.972752    2576 manager.go:319] Starting recovery of all containers
Oct 31 00:28:58.983930 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.983910    2576 manager.go:324] Recovery completed
Oct 31 00:28:58.988582 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.988454    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:58.990302 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.990269    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:58.990376 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.990321    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:58.990376 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.990332    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:58.990874 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.990863    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:58.990932 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.990865    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:28:58.990932 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.990893    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:28:58.990932 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.990910    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:28:58.991557 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.991542    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:58.991631 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.991571    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:58.991631 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.991585    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:58.993122 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.993109    2576 policy_none.go:50] "Start"
Oct 31 00:28:58.993175 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.993125    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:28:58.993175 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.993135    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:28:58.995223 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:58.995212    2576 policy_none.go:44] "Start"
Oct 31 00:28:59.035009 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.034952    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:28:59.037735 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.037721    2576 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:28:59.037785 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.037752    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:28:59.037785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.037763    2576 server.go:87] "Starting device plugin registration server"
Oct 31 00:28:59.038069 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.038057    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:28:59.038104 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.038072    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:28:59.038171 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.038155    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:28:59.038277 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.038244    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:28:59.038277 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.038256    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:28:59.038823 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.038798    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:28:59.038893 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.038858    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-97.us-west-2.compute.internal\" not found"
Oct 31 00:28:59.098396 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.098342    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 00:28:59.098396 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.098373    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:28:59.100447 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.098344    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375beccbd9e42  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.96280429 +0000 UTC m=+0.410187214,LastTimestamp:2026-10-31 00:28:58.96280429 +0000 UTC m=+0.410187214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.110246 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.110222    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:28:59.111473 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.111455    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:28:59.111536 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.111486    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:28:59.111536 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.111510    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:28:59.111593 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.111549    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 00:28:59.138624 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.138602    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:59.139340 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.139325    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:59.139436 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.139373    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:59.139436 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.139384    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:59.139436 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.139407    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.178032 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.178011    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:28:59.178100 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.178053    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8h779"
Oct 31 00:28:59.212805 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.212780    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"]
Oct 31 00:28:59.212898 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.212840    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:59.213727 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.213709    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:59.213795 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.213738    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:59.213795 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.213748    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:59.216044 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216032    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:59.216173 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216160    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.216208 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216188    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:59.216866 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216848    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:59.216926 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216872    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:59.216926 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216882    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:59.216926 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216854    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:59.216926 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216914    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:59.217041 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.216928    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:59.218959 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.218946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.219004 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.218973    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:59.219721 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.219708    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:59.219784 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.219737    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:59.219784 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.219750    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:59.240795 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.240774    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.246573 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.246548    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.250833 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.250805    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:28:59.250833 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.250748    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.251026 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.250925    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:28:59.251026 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.250974    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 00:28:59.273967 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.273916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b932d77d38edb2300c0d4d58dae2752-config\") pod \"kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"0b932d77d38edb2300c0d4d58dae2752\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.273967 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.273951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.274070 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.273970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.305665 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.305637    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 31 00:28:59.305805 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.305621    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,LastTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.305805 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.305704    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.374760 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.374732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.374854 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.374769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b932d77d38edb2300c0d4d58dae2752-config\") pod \"kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"0b932d77d38edb2300c0d4d58dae2752\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.374854 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.374786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.374854 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.374816    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b932d77d38edb2300c0d4d58dae2752-config\") pod \"kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"0b932d77d38edb2300c0d4d58dae2752\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.374854 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.374822    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.375008 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.374856    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.401006 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.400973    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="400ms"
Oct 31 00:28:59.401084 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.401012    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,LastTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.462847 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.462764    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:58.991556186 +0000 UTC m=+0.438939113,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.506834 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.506806    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:28:59.513895 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.513865    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:28:59.513895 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.513899    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:28:59.514052 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.513910    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:28:59.514052 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.513951    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.544224 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.544162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.550917 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:28:59.550901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.564235 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.564158    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,LastTimestamp:2026-10-31 00:28:58.991577493 +0000 UTC m=+0.438960419,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.613287 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.612888    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,LastTimestamp:2026-10-31 00:28:58.991591627 +0000 UTC m=+0.438974556,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.716251 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.716210    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:28:59.716437 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.716283    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bed13447ec  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated limits on system reserved cgroup /system.slice,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:59.037689836 +0000 UTC m=+0.485072760,LastTimestamp:2026-10-31 00:28:59.037689836 +0000 UTC m=+0.485072760,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.780824 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.780723    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bed14c95af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:59.039282607 +0000 UTC m=+0.486665519,LastTimestamp:2026-10-31 00:28:59.039282607 +0000 UTC m=+0.486665519,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.867104 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.867023    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="800ms"
Oct 31 00:28:59.867104 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.867009    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:59.139339628 +0000 UTC m=+0.586722553,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:28:59.944532 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:28:59.944433    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,LastTimestamp:2026-10-31 00:28:59.139377996 +0000 UTC m=+0.586760921,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.032313 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.032224    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,LastTimestamp:2026-10-31 00:28:59.139387469 +0000 UTC m=+0.586770395,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.110825 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.110795    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:00.112041 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.112022    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:00.112151 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.112056    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:00.112151 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.112069    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:00.114730 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.114641    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:59.213725172 +0000 UTC m=+0.661108097,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.117283 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.117237    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:00.117903 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.117884    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:00.117986 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.117916    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:00.117986 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.117932    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:00.117986 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.117960    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:00.184404 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.184377    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:00.184598 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.184374    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,LastTimestamp:2026-10-31 00:28:59.21374281 +0000 UTC m=+0.661125734,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.272076 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.272050    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:00.272225 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.272107    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 31 00:29:00.272401 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.272312    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,LastTimestamp:2026-10-31 00:28:59.213751592 +0000 UTC m=+0.661134517,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.346880 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.346785    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:59.216864099 +0000 UTC m=+0.664247024,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.424984 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.424839    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,LastTimestamp:2026-10-31 00:28:59.216876994 +0000 UTC m=+0.664259918,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.424984 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.424929    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:29:00.515483 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.515379    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,LastTimestamp:2026-10-31 00:28:59.216885727 +0000 UTC m=+0.664268651,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.586455 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.586333    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:59.216899071 +0000 UTC m=+0.664281995,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.680848 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.680774    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:29:00.681008 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.680788    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,LastTimestamp:2026-10-31 00:28:59.216919888 +0000 UTC m=+0.664302812,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.744469 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.744437    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="1.6s"
Oct 31 00:29:00.744469 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.744407    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,LastTimestamp:2026-10-31 00:28:59.216934435 +0000 UTC m=+0.664317361,Count:6,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.749088 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:00.748638    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8796dcae988b291465b4878c3ad6567.slice/crio-99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199 WatchSource:0}: Error finding container 99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199: Status 404 returned error can't find the container with id 99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199
Oct 31 00:29:00.749263 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:00.749246    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b932d77d38edb2300c0d4d58dae2752.slice/crio-389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d WatchSource:0}: Error finding container 389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d: Status 404 returned error can't find the container with id 389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d
Oct 31 00:29:00.754573 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:00.754560    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:29:00.817412 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.817371    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 00:29:00.817558 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.817399    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:59.219721023 +0000 UTC m=+0.667103959,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.896615 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.896530    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61928d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990326413 +0000 UTC m=+0.437709338,LastTimestamp:2026-10-31 00:28:59.219743974 +0000 UTC m=+0.667126898,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:00.967097 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:00.966970    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece61b84b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990336075 +0000 UTC m=+0.437718999,LastTimestamp:2026-10-31 00:28:59.219753699 +0000 UTC m=+0.667136623,Count:7,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:01.046703 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:01.046619    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-97.us-west-2.compute.internal.18e375bece613815  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-97.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-97.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:28:58.990303253 +0000 UTC m=+0.437686178,LastTimestamp:2026-10-31 00:28:59.513888727 +0000 UTC m=+0.961271652,Count:8,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:01.072867 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.072841    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:01.073758 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.073742    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:01.073839 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.073769    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:01.073839 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.073780    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:01.073839 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.073809    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:01.108997 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:01.108914    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal.18e375bf379102bd  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal,UID:0b932d77d38edb2300c0d4d58dae2752,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\",Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:00.755043005 +0000 UTC m=+2.202425927,LastTimestamp:2026-10-31 00:29:00.755043005 +0000 UTC m=+2.202425927,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:01.117914 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.116648    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerStarted","Data":"99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199"}
Oct 31 00:29:01.119077 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.119055    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"0b932d77d38edb2300c0d4d58dae2752","Type":"ContainerStarted","Data":"389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d"}
Oct 31 00:29:01.184507 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:01.184476    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:01.184658 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:01.184565    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal.18e375bf37a5fc36  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal,UID:e8796dcae988b291465b4878c3ad6567,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulling,Message:Pulling image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\",Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:00.75641759 +0000 UTC m=+2.203800503,LastTimestamp:2026-10-31 00:29:00.75641759 +0000 UTC m=+2.203800503,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:01.247522 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:01.247450    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:02.143289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.143256    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:02.290885 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.290850    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:02.292262 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.292242    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:02.292411 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.292277    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:02.292411 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.292290    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:02.433938 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:02.433852    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="3.2s"
Oct 31 00:29:02.711550 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:02.711478    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 31 00:29:02.711550 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:02.711479    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal.18e375bfa62fa84c  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal,UID:e8796dcae988b291465b4878c3ad6567,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d017b7d199b1d54e889bb1eb6e572d8e6d94ad92610894009ee0562c2a5c0f51\" in 1.854s (1.854s including waiting). Image size: 272339155 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:02.610933836 +0000 UTC m=+4.058316767,LastTimestamp:2026-10-31 00:29:02.610933836 +0000 UTC m=+4.058316767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:02.848238 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.848110    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:02.849119 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:02.849097    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 00:29:02.849210 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.849197    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:02.849250 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.849222    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:02.849250 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.849231    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:02.849316 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:02.849256    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:02.998940 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:02.998762    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal.18e375bfb2c7ea76  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal,UID:0b932d77d38edb2300c0d4d58dae2752,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Pulled,Message:Successfully pulled image \"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3a679d80eb36ca7db888d4fbd1fdf16b0997a7c7a107936396b7a57d8b199f4e\" in 2.067s (2.067s including waiting). Image size: 294084393 bytes.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:02.822238838 +0000 UTC m=+4.269621748,LastTimestamp:2026-10-31 00:29:02.822238838 +0000 UTC m=+4.269621748,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:03.123483 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.123453    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"0b932d77d38edb2300c0d4d58dae2752","Type":"ContainerStarted","Data":"7f4c98fe5dd5b033d354ae3a78949a43f6290ab2673dc7865dbf458810fc7f9f"}
Oct 31 00:29:03.123628 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.123466    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:03.124447 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.124426    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:03.124546 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.124455    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:03.124546 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.124464    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:03.124692 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.124681    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:03.124752 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.124735    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerStarted","Data":"6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e"}
Oct 31 00:29:03.124811 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.124800    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:03.125487 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.125474    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:03.125562 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.125497    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:03.125562 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.125508    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:03.125649 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.125639    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:03.137401 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.137383    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:03.137482 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.137361    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal.18e375bfb39b8287  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal,UID:e8796dcae988b291465b4878c3ad6567,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:02.836105863 +0000 UTC m=+4.283488793,LastTimestamp:2026-10-31 00:29:02.836105863 +0000 UTC m=+4.283488793,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:03.279911 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:03.279827    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:03.280388 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.279868    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-machine-config-operator\"" event="&Event{ObjectMeta:{kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal.18e375bfb41f64ad  openshift-machine-config-operator    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:openshift-machine-config-operator,Name:kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal,UID:e8796dcae988b291465b4878c3ad6567,APIVersion:v1,ResourceVersion:,FieldPath:spec.initContainers{setup},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:02.844748973 +0000 UTC m=+4.292131884,LastTimestamp:2026-10-31 00:29:02.844748973 +0000 UTC m=+4.292131884,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:03.417581 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.417551    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:29:03.417712 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.417535    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal.18e375bfb49ea10b  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal,UID:0b932d77d38edb2300c0d4d58dae2752,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Created,Message:Container created,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:02.853087499 +0000 UTC m=+4.300470413,LastTimestamp:2026-10-31 00:29:02.853087499 +0000 UTC m=+4.300470413,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:03.566643 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.566525    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"kube-system\"" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal.18e375bfb508eb8a  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal,UID:0b932d77d38edb2300c0d4d58dae2752,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Started,Message:Container started,Source:EventSource{Component:kubelet,Host:ip-10-0-131-97.us-west-2.compute.internal,},FirstTimestamp:2026-10-31 00:29:02.860053386 +0000 UTC m=+4.307436313,LastTimestamp:2026-10-31 00:29:02.860053386 +0000 UTC m=+4.307436313,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-97.us-west-2.compute.internal,}"
Oct 31 00:29:03.968600 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:03.968530    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:29:04.125878 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.125856    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:04.126001 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.125857    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:04.126642 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.126622    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:04.126755 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.126657    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:04.126755 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.126667    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:04.126755 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.126622    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:04.126755 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.126731    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:04.126755 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.126743    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:04.126989 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:04.126891    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:04.126989 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:04.126928    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:04.251341 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:04.251260    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:05.262457 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:05.262425    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:05.784319 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:05.784292    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="6.4s"
Oct 31 00:29:06.248365 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.248269    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:06.337457 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.337426    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:06.338462 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.338438    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:06.338567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.338470    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:06.338567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.338482    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:06.338567 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.338508    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:06.410529 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:06.410491    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 00:29:06.511023 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:06.510954    2576 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-131-97.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:06.846100 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.846029    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:06.848045 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.848021    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:06.848162 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.848054    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:06.848162 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:06.848065    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:07.251403 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:07.251307    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:29:07.475921 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:07.475893    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8h779"
Oct 31 00:29:07.902754 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:07.902719    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 00:29:08.300561 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.300484    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:08.361469 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.361446    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:08.463630 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.463610    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:08.478948 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.478924    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:24:07 +0000 UTC" deadline="2028-06-26 22:21:38.509354257 +0000 UTC"
Oct 31 00:29:08.478948 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.478947    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14517h52m30.030410403s"
Oct 31 00:29:08.770311 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.770239    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:08.770311 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:08.770264    2576 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:08.852411 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.852388    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:08.913291 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:08.913269    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:09.015504 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.015480    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:09.039075 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:09.039013    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-97.us-west-2.compute.internal\" not found"
Oct 31 00:29:09.057790 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.057768    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:09.189913 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.189889    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:09.320395 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.320330    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:09.320395 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:09.320385    2576 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:09.621386 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.621311    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:09.684730 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.684709    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:09.789931 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.789912    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:09.904149 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.904080    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:09.955539 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:09.955523    2576 apiserver.go:52] "Watching apiserver"
Oct 31 00:29:10.011241 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:10.011219    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:29:10.011302 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:10.011258    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=[]
Oct 31 00:29:10.072132 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:10.072107    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:29:10.108669 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:10.108644    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:10.108669 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:10.108672    2576 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:12.270248 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.270218    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:12.284040 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:12.284014    2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:12.331241 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.331222    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:12.436600 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.436580    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:12.742384 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.742291    2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:12.742384 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:12.742317    2576 csi_plugin.go:399] Failed to initialize CSINode: error updating CSINode annotation: timed out waiting for the condition; caused by: nodes "ip-10-0-131-97.us-west-2.compute.internal" not found
Oct 31 00:29:12.911395 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.911369    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:12.912362 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.912339    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:12.912451 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.912377    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:12.912451 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.912387    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:12.912451 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.912413    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:12.966982 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.966965    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:12.971849 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.971830    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:12.979334 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:12.979311    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 00:29:13.015911 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.015854    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-j9jsc"]
Oct 31 00:29:13.018213 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.018198    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.023596 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.023578    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-gxlg7"]
Oct 31 00:29:13.026159 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.026139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.061399 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.061367    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:29:13.061494 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.061432    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:13.061494 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.061464    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"]
Oct 31 00:29:13.063843 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.063826    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-wjjv6"]
Oct 31 00:29:13.063953 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.063941    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.066026 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.066002    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"]
Oct 31 00:29:13.066026 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.066026    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-cntts"]
Oct 31 00:29:13.066150 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.066098    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.068142 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.068124    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-mr44j"]
Oct 31 00:29:13.068268 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.068250    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.070443 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.070418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:13.070819 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.070782    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:13.072176 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.071297    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-2fk4z"]
Oct 31 00:29:13.074160 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.074139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:13.074250 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.074206    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:13.082167 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.082149    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:29:13.082236 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.082191    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzctz\""
Oct 31 00:29:13.108138 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108116    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:29:13.108239 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108216    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:29:13.108466 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108451    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:29:13.108539 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108516    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:29:13.108667 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108653    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-2rxnb"]
Oct 31 00:29:13.108731 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108682    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:29:13.108731 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108681    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:29:13.108808 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.108786    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:29:13.111070 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.111056    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-jh76v"]
Oct 31 00:29:13.111183 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.111169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.113457 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.113438    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.115525 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.115509    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-gmcv9"]
Oct 31 00:29:13.117748 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.117735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.118176 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.118164    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-vx9mh"]
Oct 31 00:29:13.120717 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.120702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.129178 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.129164    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:29:13.129401 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.129387    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"]
Oct 31 00:29:13.151614 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:13.151681 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tz66\" (UniqueName: \"kubernetes.io/projected/4d53981b-d854-4b47-b1f1-f46b0c810830-kube-api-access-4tz66\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:13.151681 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151633    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-kubelet\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.151681 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.151778 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-env-overrides\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.151778 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxww8\" (UniqueName: \"kubernetes.io/projected/c36f1c1a-8072-409c-ba64-929fe596778b-kube-api-access-hxww8\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.151836 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsshs\" (UniqueName: \"kubernetes.io/projected/eaaec7db-74f2-49ac-a56b-198f78680fce-kube-api-access-zsshs\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.151836 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-netd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.151893 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.151893 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.151893 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaaec7db-74f2-49ac-a56b-198f78680fce-iptables-alerter-script\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.151971 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:13.151971 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-systemd-units\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152024 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.151976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-netns\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152024 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152076 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-node-log\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152076 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-log-socket\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152076 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152076 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.152182 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.152182 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-slash\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152182 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-etc-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152260 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm2hb\" (UniqueName: \"kubernetes.io/projected/01df07af-c80e-4ae5-950d-87cc511cbda7-kube-api-access-bm2hb\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152260 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-device-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.152260 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.152335 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36f1c1a-8072-409c-ba64-929fe596778b-tmp-dir\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.152335 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-ovn\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152335 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-config\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152335 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01df07af-c80e-4ae5-950d-87cc511cbda7-ovn-node-metrics-cert\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152335 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7bvw\" (UniqueName: \"kubernetes.io/projected/3d584253-d14a-48e9-bfe6-563acb954e1f-kube-api-access-d7bvw\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.152490 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c36f1c1a-8072-409c-ba64-929fe596778b-hosts-file\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.152490 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-host\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.152490 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-systemd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152490 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-var-lib-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152490 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152482    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-bin\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152623 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-script-lib\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.152623 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152529    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-serviceca\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.152623 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8zh\" (UniqueName: \"kubernetes.io/projected/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-kube-api-access-cl8zh\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.152623 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.152572    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaec7db-74f2-49ac-a56b-198f78680fce-host-slash\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.155515 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.155495    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:29:13.155635 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.155524    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:29:13.155635 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.155565    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:29:13.155635 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.155568    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:29:13.155635 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.155594    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:29:13.155838 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.155765    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hrxqv"]
Oct 31 00:29:13.156008 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.155994    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:29:13.156058 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.156024    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:29:13.158576 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.158561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.158638 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.158623    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:13.170823 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.170802    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:29:13.170911 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.170863    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2nqz7"
Oct 31 00:29:13.172275 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.172250    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:29:13.172375 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.172284    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:29:13.172375 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.172288    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:29:13.172524 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.172510    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:29:13.172570 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.172528    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:29:13.172570 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.172544    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:29:13.172570 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.172555    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:29:13.173393 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.173380    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-klgwd\""
Oct 31 00:29:13.173514 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.173499    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:29:13.176432 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.176420    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:29:13.176499 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.176466    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:29:13.176499 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.176488    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:29:13.176499 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.176495    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:29:13.176609 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.176510    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s5c24\""
Oct 31 00:29:13.176609 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.176514    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2sfqr\""
Oct 31 00:29:13.176609 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.176561    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6xpx\""
Oct 31 00:29:13.182760 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.182746    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bv698\""
Oct 31 00:29:13.182760 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.182753    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4l8km\""
Oct 31 00:29:13.182760 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.182751    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5b488\""
Oct 31 00:29:13.182896 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.182830    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l8m77\""
Oct 31 00:29:13.253636 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsshs\" (UniqueName: \"kubernetes.io/projected/eaaec7db-74f2-49ac-a56b-198f78680fce-kube-api-access-zsshs\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.253636 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-sys\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.253785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cnibin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.253785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253667    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-bin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.253785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.253785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaaec7db-74f2-49ac-a56b-198f78680fce-iptables-alerter-script\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.253785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:13.253785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253766    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-host\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254013 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-etc-tuned\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254013 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-multus-certs\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.254013 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253846    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-kubelet-config\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.254013 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-lib-modules\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254013 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-netns\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254013 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254013 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.253984    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-netns\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254068    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-conf\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254159    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-netns\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-systemd\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-slash\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaaec7db-74f2-49ac-a56b-198f78680fce-iptables-alerter-script\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.254315 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254309    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-slash\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-etc-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2hb\" (UniqueName: \"kubernetes.io/projected/01df07af-c80e-4ae5-950d-87cc511cbda7-kube-api-access-bm2hb\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-socket-dir-parent\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254425    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-etc-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh6c4\" (UniqueName: \"kubernetes.io/projected/7c477db4-252e-4236-9ddd-79307115124f-kube-api-access-nh6c4\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-kubernetes\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-script-lib\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-serviceca\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8zh\" (UniqueName: \"kubernetes.io/projected/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-kube-api-access-cl8zh\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysconfig\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l27tt\" (UniqueName: \"kubernetes.io/projected/858636bd-15aa-47e2-95ce-daaf7cffb8d0-kube-api-access-l27tt\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-tmp\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-multus\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.254704    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:13.254718 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.254769    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs podName:4d53981b-d854-4b47-b1f1-f46b0c810830 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:13.754750405 +0000 UTC m=+15.202133333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs") pod "network-metrics-daemon-2fk4z" (UID: "4d53981b-d854-4b47-b1f1-f46b0c810830") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-netd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254890    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-netd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-dbus\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-systemd-units\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.254998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-node-log\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255007    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-systemd-units\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255021    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255029    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-log-socket\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255047    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-node-log\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-log-socket\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.255485 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36f1c1a-8072-409c-ba64-929fe596778b-tmp-dir\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-modprobe-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-device-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255199    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-system-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-k8s-cni-cncf-io\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255255    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-device-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-conf-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxbsz\" (UniqueName: \"kubernetes.io/projected/0aeb97e6-8dcd-4896-8a32-424681fcfc47-kube-api-access-qxbsz\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-var-lib-kubelet\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-ovn\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255416    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36f1c1a-8072-409c-ba64-929fe596778b-tmp-dir\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255438    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-ovn\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-config\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01df07af-c80e-4ae5-950d-87cc511cbda7-ovn-node-metrics-cert\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-script-lib\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bvw\" (UniqueName: \"kubernetes.io/projected/3d584253-d14a-48e9-bfe6-563acb954e1f-kube-api-access-d7bvw\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c36f1c1a-8072-409c-ba64-929fe596778b-hosts-file\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255715    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-serviceca\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-host\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-run\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-systemd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-var-lib-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c36f1c1a-8072-409c-ba64-929fe596778b-hosts-file\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-systemd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-bin\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-host\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cni-binary-copy\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255852    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-bin\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-var-lib-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-kubelet\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255852    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-etc-kubernetes\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255903    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-config\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.256785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255917    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fd43-f514-4533-9523-0f8f97930639-konnectivity-ca\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-system-cni-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.255989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaec7db-74f2-49ac-a56b-198f78680fce-host-slash\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256041    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaec7db-74f2-49ac-a56b-198f78680fce-host-slash\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-daemon-config\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b07fd43-f514-4533-9523-0f8f97930639-agent-certs\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cnibin\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tz66\" (UniqueName: \"kubernetes.io/projected/4d53981b-d854-4b47-b1f1-f46b0c810830-kube-api-access-4tz66\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-hostroot\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-kubelet\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.257245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.257672 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256463    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-env-overrides\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.257672 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxww8\" (UniqueName: \"kubernetes.io/projected/c36f1c1a-8072-409c-ba64-929fe596778b-kube-api-access-hxww8\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.257672 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-kubelet\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.257672 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.256499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.257672 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.257157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-env-overrides\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.259341 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.259322    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01df07af-c80e-4ae5-950d-87cc511cbda7-ovn-node-metrics-cert\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.263831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.263776    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal" podStartSLOduration=0.263763395 podStartE2EDuration="263.763395ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:13.263306663 +0000 UTC m=+14.710689589" watchObservedRunningTime="2026-10-31 00:29:13.263763395 +0000 UTC m=+14.711146330"
Oct 31 00:29:13.311816 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.311769    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:29:13.311816 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.311787    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:29:13.311816 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.311797    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bm7dt for pod openshift-network-diagnostics/network-check-target-mr44j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:13.312167 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.311838    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt podName:a5f64145-a6c9-4d5c-8951-348943d7806c nodeName:}" failed. No retries permitted until 2026-10-31 00:29:13.811825514 +0000 UTC m=+15.259208450 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bm7dt" (UniqueName: "kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt") pod "network-check-target-mr44j" (UID: "a5f64145-a6c9-4d5c-8951-348943d7806c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:13.314677 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.314655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8zh\" (UniqueName: \"kubernetes.io/projected/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-kube-api-access-cl8zh\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.314765 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.314686    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2hb\" (UniqueName: \"kubernetes.io/projected/01df07af-c80e-4ae5-950d-87cc511cbda7-kube-api-access-bm2hb\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.314765 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.314658    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsshs\" (UniqueName: \"kubernetes.io/projected/eaaec7db-74f2-49ac-a56b-198f78680fce-kube-api-access-zsshs\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.320016 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.319991    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxww8\" (UniqueName: \"kubernetes.io/projected/c36f1c1a-8072-409c-ba64-929fe596778b-kube-api-access-hxww8\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.320922 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.320907    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tz66\" (UniqueName: \"kubernetes.io/projected/4d53981b-d854-4b47-b1f1-f46b0c810830-kube-api-access-4tz66\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:13.321002 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.320983    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bvw\" (UniqueName: \"kubernetes.io/projected/3d584253-d14a-48e9-bfe6-563acb954e1f-kube-api-access-d7bvw\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.328316 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.328302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:13.348312 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.348296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:13.356360 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.356328    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01df07af_c80e_4ae5_950d_87cc511cbda7.slice/crio-06a421cfd426cf060983d93846152acb56e9cb14ba8f96e860b5cbb13ce391ec WatchSource:0}: Error finding container 06a421cfd426cf060983d93846152acb56e9cb14ba8f96e860b5cbb13ce391ec: Status 404 returned error can't find the container with id 06a421cfd426cf060983d93846152acb56e9cb14ba8f96e860b5cbb13ce391ec
Oct 31 00:29:13.357055 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-dbus\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.357138 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-modprobe-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.357193 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-modprobe-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.357193 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-system-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-k8s-cni-cncf-io\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-conf-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357180    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-dbus\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.357289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbsz\" (UniqueName: \"kubernetes.io/projected/0aeb97e6-8dcd-4896-8a32-424681fcfc47-kube-api-access-qxbsz\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357255    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-system-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357265    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-k8s-cni-cncf-io\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357289 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357273    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-var-lib-kubelet\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-run\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357314    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-var-lib-kubelet\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cni-binary-copy\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357334    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-run\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-conf-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357363    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-kubelet\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357387    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-kubelet\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-etc-kubernetes\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fd43-f514-4533-9523-0f8f97930639-konnectivity-ca\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357471    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-etc-kubernetes\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-system-cni-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-daemon-config\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357573    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-system-cni-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b07fd43-f514-4533-9523-0f8f97930639-agent-certs\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.357616 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cnibin\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357689    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-hostroot\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357733    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357757    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-sys\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cnibin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-bin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-host\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357869    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cni-binary-copy\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-etc-tuned\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-multus-certs\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-kubelet-config\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-lib-modules\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357956    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fd43-f514-4533-9523-0f8f97930639-konnectivity-ca\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-conf\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.358454 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-netns\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-systemd\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-socket-dir-parent\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358052    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cnibin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.357639    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358066    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6c4\" (UniqueName: \"kubernetes.io/projected/7c477db4-252e-4236-9ddd-79307115124f-kube-api-access-nh6c4\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358095    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-kubernetes\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358104    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-daemon-config\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysconfig\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358138    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27tt\" (UniqueName: \"kubernetes.io/projected/858636bd-15aa-47e2-95ce-daaf7cffb8d0-kube-api-access-l27tt\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysconfig\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-tmp\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-multus\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.358251    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358283    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358309    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-bin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358325    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-sys\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.358366    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret podName:2a1760d4-777c-4853-bbbf-0a9afae8f45f nodeName:}" failed. No retries permitted until 2026-10-31 00:29:13.858331892 +0000 UTC m=+15.305714820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret") pod "global-pull-secret-syncer-hrxqv" (UID: "2a1760d4-777c-4853-bbbf-0a9afae8f45f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358385    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-kubernetes\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-host\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358431    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-multus-certs\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358470    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-netns\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358489    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-kubelet-config\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cnibin\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358583    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-lib-modules\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-conf\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358710    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-systemd\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358719    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-multus\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358761    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-socket-dir-parent\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.358785    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-hostroot\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.359084    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.359763 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.359331    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.360511 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.360494    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b07fd43-f514-4533-9523-0f8f97930639-agent-certs\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.360606 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.360585    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-etc-tuned\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.361014 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.360997    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-tmp\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.378451 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.378434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:13.381555 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.381541    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:13.385254 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.385242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:13.389207 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.389185    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d584253_d14a_48e9_bfe6_563acb954e1f.slice/crio-6843b645ac49edfc71489a9ed5110c48402c455d35ebb81e86185affa9e03652 WatchSource:0}: Error finding container 6843b645ac49edfc71489a9ed5110c48402c455d35ebb81e86185affa9e03652: Status 404 returned error can't find the container with id 6843b645ac49edfc71489a9ed5110c48402c455d35ebb81e86185affa9e03652
Oct 31 00:29:13.389830 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.389810    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc36f1c1a_8072_409c_ba64_929fe596778b.slice/crio-5f898bf492e7ce0e4cd1003a6c9536d8f2f903bc7d8d29cc122aa5b41dcefe8e WatchSource:0}: Error finding container 5f898bf492e7ce0e4cd1003a6c9536d8f2f903bc7d8d29cc122aa5b41dcefe8e: Status 404 returned error can't find the container with id 5f898bf492e7ce0e4cd1003a6c9536d8f2f903bc7d8d29cc122aa5b41dcefe8e
Oct 31 00:29:13.393807 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.393789    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaec7db_74f2_49ac_a56b_198f78680fce.slice/crio-1223b71cf0a603c934d3f9c07558f7c8a570f784f970a554124ed3bc3020caae WatchSource:0}: Error finding container 1223b71cf0a603c934d3f9c07558f7c8a570f784f970a554124ed3bc3020caae: Status 404 returned error can't find the container with id 1223b71cf0a603c934d3f9c07558f7c8a570f784f970a554124ed3bc3020caae
Oct 31 00:29:13.426559 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.426534    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbsz\" (UniqueName: \"kubernetes.io/projected/0aeb97e6-8dcd-4896-8a32-424681fcfc47-kube-api-access-qxbsz\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.428645 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.428623    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6c4\" (UniqueName: \"kubernetes.io/projected/7c477db4-252e-4236-9ddd-79307115124f-kube-api-access-nh6c4\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.429123 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.429108    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27tt\" (UniqueName: \"kubernetes.io/projected/858636bd-15aa-47e2-95ce-daaf7cffb8d0-kube-api-access-l27tt\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.432056 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.432042    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:13.441209 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.441186    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b07fd43_f514_4533_9523_0f8f97930639.slice/crio-32adff65999d9cdc6af88443bc0074345790dc83bd70c2344f52fe4279ceff50 WatchSource:0}: Error finding container 32adff65999d9cdc6af88443bc0074345790dc83bd70c2344f52fe4279ceff50: Status 404 returned error can't find the container with id 32adff65999d9cdc6af88443bc0074345790dc83bd70c2344f52fe4279ceff50
Oct 31 00:29:13.447429 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.447410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:13.456716 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.456693    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858636bd_15aa_47e2_95ce_daaf7cffb8d0.slice/crio-a8d2e85eca66a1429f17d4cbc5e4f7613bf3e15d0017439c4a2a7991ef0bdd08 WatchSource:0}: Error finding container a8d2e85eca66a1429f17d4cbc5e4f7613bf3e15d0017439c4a2a7991ef0bdd08: Status 404 returned error can't find the container with id a8d2e85eca66a1429f17d4cbc5e4f7613bf3e15d0017439c4a2a7991ef0bdd08
Oct 31 00:29:13.722066 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.721994    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:13.726866 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.726850    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jh76v"
Oct 31 00:29:13.733152 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.733127    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c477db4_252e_4236_9ddd_79307115124f.slice/crio-d9bcfabc7689b8c50cc6e7b71105ca1d77c69b16903770bfaf1025b3f2a02dbd WatchSource:0}: Error finding container d9bcfabc7689b8c50cc6e7b71105ca1d77c69b16903770bfaf1025b3f2a02dbd: Status 404 returned error can't find the container with id d9bcfabc7689b8c50cc6e7b71105ca1d77c69b16903770bfaf1025b3f2a02dbd
Oct 31 00:29:13.736022 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:13.736000    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aeb97e6_8dcd_4896_8a32_424681fcfc47.slice/crio-87caaa89b01db633825a45da0a845742cec1b15a776d8180df82f7dc6e9fc728 WatchSource:0}: Error finding container 87caaa89b01db633825a45da0a845742cec1b15a776d8180df82f7dc6e9fc728: Status 404 returned error can't find the container with id 87caaa89b01db633825a45da0a845742cec1b15a776d8180df82f7dc6e9fc728
Oct 31 00:29:13.760383 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.760367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:13.760519 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.760504    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:13.760562 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.760554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs podName:4d53981b-d854-4b47-b1f1-f46b0c810830 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:14.760541709 +0000 UTC m=+16.207924641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs") pod "network-metrics-daemon-2fk4z" (UID: "4d53981b-d854-4b47-b1f1-f46b0c810830") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:13.861177 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.861150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:13.861335 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:13.861193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:13.861335 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.861284    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:13.861335 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.861302    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:29:13.861335 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.861317    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:29:13.861335 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.861327    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bm7dt for pod openshift-network-diagnostics/network-check-target-mr44j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:13.861531 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.861362    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret podName:2a1760d4-777c-4853-bbbf-0a9afae8f45f nodeName:}" failed. No retries permitted until 2026-10-31 00:29:14.861330993 +0000 UTC m=+16.308713906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret") pod "global-pull-secret-syncer-hrxqv" (UID: "2a1760d4-777c-4853-bbbf-0a9afae8f45f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:13.861531 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:13.861389    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt podName:a5f64145-a6c9-4d5c-8951-348943d7806c nodeName:}" failed. No retries permitted until 2026-10-31 00:29:14.861374186 +0000 UTC m=+16.308757100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm7dt" (UniqueName: "kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt") pod "network-check-target-mr44j" (UID: "a5f64145-a6c9-4d5c-8951-348943d7806c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:14.112524 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.112456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:14.112649 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.112561    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:14.112649 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.112619    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:14.112734 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.112716    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:14.139290 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.139264    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wjjv6" event={"ID":"c36f1c1a-8072-409c-ba64-929fe596778b","Type":"ContainerStarted","Data":"5f898bf492e7ce0e4cd1003a6c9536d8f2f903bc7d8d29cc122aa5b41dcefe8e"}
Oct 31 00:29:14.140260 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.140240    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerStarted","Data":"6843b645ac49edfc71489a9ed5110c48402c455d35ebb81e86185affa9e03652"}
Oct 31 00:29:14.141529 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.141511    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"06a421cfd426cf060983d93846152acb56e9cb14ba8f96e860b5cbb13ce391ec"}
Oct 31 00:29:14.142884 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.142870    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal_e8796dcae988b291465b4878c3ad6567/setup/0.log"
Oct 31 00:29:14.142936 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.142898    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8796dcae988b291465b4878c3ad6567" containerID="6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e" exitCode=1
Oct 31 00:29:14.142979 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.142946    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerDied","Data":"6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e"}
Oct 31 00:29:14.144001 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.143985    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jh76v" event={"ID":"0aeb97e6-8dcd-4896-8a32-424681fcfc47","Type":"ContainerStarted","Data":"87caaa89b01db633825a45da0a845742cec1b15a776d8180df82f7dc6e9fc728"}
Oct 31 00:29:14.145018 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.145000    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rxnb" event={"ID":"7c477db4-252e-4236-9ddd-79307115124f","Type":"ContainerStarted","Data":"d9bcfabc7689b8c50cc6e7b71105ca1d77c69b16903770bfaf1025b3f2a02dbd"}
Oct 31 00:29:14.146151 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.146132    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerStarted","Data":"a8d2e85eca66a1429f17d4cbc5e4f7613bf3e15d0017439c4a2a7991ef0bdd08"}
Oct 31 00:29:14.147254 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.147232    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmcv9" event={"ID":"7b07fd43-f514-4533-9523-0f8f97930639","Type":"ContainerStarted","Data":"32adff65999d9cdc6af88443bc0074345790dc83bd70c2344f52fe4279ceff50"}
Oct 31 00:29:14.148272 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.148252    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cntts" event={"ID":"eaaec7db-74f2-49ac-a56b-198f78680fce","Type":"ContainerStarted","Data":"1223b71cf0a603c934d3f9c07558f7c8a570f784f970a554124ed3bc3020caae"}
Oct 31 00:29:14.149234 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.149216    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j9jsc" event={"ID":"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e","Type":"ContainerStarted","Data":"d392df2bf42859cf3facac13cbe5f1b870941235d40ddb1482e793fb18b275d9"}
Oct 31 00:29:14.174313 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.174258    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:24:13 +0000 UTC" deadline="2028-06-01 01:26:53.090490364 +0000 UTC"
Oct 31 00:29:14.174313 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.174312    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13896h57m38.916182474s"
Oct 31 00:29:14.771626 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.771589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:14.772100 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.771729    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:14.772100 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.771787    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs podName:4d53981b-d854-4b47-b1f1-f46b0c810830 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:16.771769914 +0000 UTC m=+18.219152831 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs") pod "network-metrics-daemon-2fk4z" (UID: "4d53981b-d854-4b47-b1f1-f46b0c810830") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:14.872812 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.872773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:14.872994 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:14.872830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:14.873081 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.873065    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:29:14.873145 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.873092    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:29:14.873145 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.873105    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bm7dt for pod openshift-network-diagnostics/network-check-target-mr44j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:14.873246 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.873164    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt podName:a5f64145-a6c9-4d5c-8951-348943d7806c nodeName:}" failed. No retries permitted until 2026-10-31 00:29:16.873144954 +0000 UTC m=+18.320527890 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm7dt" (UniqueName: "kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt") pod "network-check-target-mr44j" (UID: "a5f64145-a6c9-4d5c-8951-348943d7806c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:14.873786 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.873455    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:14.873786 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:14.873503    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret podName:2a1760d4-777c-4853-bbbf-0a9afae8f45f nodeName:}" failed. No retries permitted until 2026-10-31 00:29:16.873489763 +0000 UTC m=+18.320872689 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret") pod "global-pull-secret-syncer-hrxqv" (UID: "2a1760d4-777c-4853-bbbf-0a9afae8f45f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:15.115079 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.114957    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:15.115292 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:15.115079    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:15.175323 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.175288    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:24:13 +0000 UTC" deadline="2028-04-16 21:27:49.276472315 +0000 UTC"
Oct 31 00:29:15.175323 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.175321    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12812h58m34.101155331s"
Oct 31 00:29:15.180918 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.180896    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal_e8796dcae988b291465b4878c3ad6567/setup/0.log"
Oct 31 00:29:15.181031 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.180940    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8796dcae988b291465b4878c3ad6567" containerID="4bbc3bf450777b45be3094e5fc80092834b483e254753859328a191a79454bdc" exitCode=0
Oct 31 00:29:15.181031 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.180981    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerDied","Data":"4bbc3bf450777b45be3094e5fc80092834b483e254753859328a191a79454bdc"}
Oct 31 00:29:15.181137 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.181036    2576 scope.go:122] "RemoveContainer" containerID="6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e"
Oct 31 00:29:15.181381 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:15.181361    2576 scope.go:122] "RemoveContainer" containerID="6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e"
Oct 31 00:29:15.271098 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:15.271055    2576 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal_openshift-machine-config-operator_e8796dcae988b291465b4878c3ad6567_0 in pod sandbox 99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199 from index: no such id: '6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e'" containerID="6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e"
Oct 31 00:29:15.271254 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:15.271138    2576 kuberuntime_container.go:987] "Unhandled Error" err="failed to remove pod init container \"setup\": rpc error: code = Unknown desc = failed to delete container k8s_setup_kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal_openshift-machine-config-operator_e8796dcae988b291465b4878c3ad6567_0 in pod sandbox 99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199 from index: no such id: '6f93f956ff884e11afaf645efeb247975462ae4a620ab0c7706e3fce44ef1f4e'; Skipping pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal_openshift-machine-config-operator(e8796dcae988b291465b4878c3ad6567)\"" logger="UnhandledError"
Oct 31 00:29:16.112943 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:16.112720    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:16.113426 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.113049    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:16.113514 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:16.113428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:16.113571 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.113514    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:16.191231 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:16.191191    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerStarted","Data":"82503f93adfec75ac05daecb881ea73b7d2d9555a3f5843dca9ceb8f8a07bb63"}
Oct 31 00:29:16.302270 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:16.302211    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" podStartSLOduration=3.302192071 podStartE2EDuration="3.302192071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:16.301667607 +0000 UTC m=+17.749050542" watchObservedRunningTime="2026-10-31 00:29:16.302192071 +0000 UTC m=+17.749575006"
Oct 31 00:29:16.790053 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:16.789546    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:16.790053 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.789690    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:16.790053 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.789754    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs podName:4d53981b-d854-4b47-b1f1-f46b0c810830 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:20.789736502 +0000 UTC m=+22.237119433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs") pod "network-metrics-daemon-2fk4z" (UID: "4d53981b-d854-4b47-b1f1-f46b0c810830") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:16.890203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:16.890255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.890448    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.890462    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.890470    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bm7dt for pod openshift-network-diagnostics/network-check-target-mr44j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.890513    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt podName:a5f64145-a6c9-4d5c-8951-348943d7806c nodeName:}" failed. No retries permitted until 2026-10-31 00:29:20.890501689 +0000 UTC m=+22.337884614 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm7dt" (UniqueName: "kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt") pod "network-check-target-mr44j" (UID: "a5f64145-a6c9-4d5c-8951-348943d7806c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.890579    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:16.892410 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:16.890599    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret podName:2a1760d4-777c-4853-bbbf-0a9afae8f45f nodeName:}" failed. No retries permitted until 2026-10-31 00:29:20.890592783 +0000 UTC m=+22.337975708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret") pod "global-pull-secret-syncer-hrxqv" (UID: "2a1760d4-777c-4853-bbbf-0a9afae8f45f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:17.120555 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:17.120494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:17.121070 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:17.120633    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:17.198773 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:17.198689    2576 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="049ded342751a7a0693a424c1b7c38c7c0ef81e18bd0dd528b7eb8ad934d791e" exitCode=0
Oct 31 00:29:17.198773 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:17.198751    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerDied","Data":"049ded342751a7a0693a424c1b7c38c7c0ef81e18bd0dd528b7eb8ad934d791e"}
Oct 31 00:29:17.986605 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:17.986575    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:18.112787 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:18.112722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:18.112921 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:18.112825    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:18.112921 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:18.112723    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:18.112921 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:18.112883    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:18.204488 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:18.204411    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gmcv9" event={"ID":"7b07fd43-f514-4533-9523-0f8f97930639","Type":"ContainerStarted","Data":"cbe4e1ebf6384e03f8616d7195d5cb1dab2966a954a7d18476960b64e9367a78"}
Oct 31 00:29:18.433424 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:18.433312    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:18.434344 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:18.434100    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:18.546148 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:18.545838    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gmcv9" podStartSLOduration=1.131276218 podStartE2EDuration="5.545819709s" totalImagesPullingTime="4.414543491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:18.3162884 +0000 UTC m=+19.763671346" watchObservedRunningTime="2026-10-31 00:29:18.545819709 +0000 UTC m=+19.993202643"
Oct 31 00:29:19.115316 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:19.114838    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:19.115316 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:19.114928    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:19.210408 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:19.210319    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-j9jsc" event={"ID":"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e","Type":"ContainerStarted","Data":"725bda0074aa1246e14ac9032f363d7703d64eb9042cc8cfe8ac0376cb4c94b1"}
Oct 31 00:29:19.321988 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:19.320879    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-j9jsc" podStartSLOduration=2.253343098 podStartE2EDuration="7.320864459s" totalImagesPullingTime="5.067521361s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:19.320788326 +0000 UTC m=+20.768171260" watchObservedRunningTime="2026-10-31 00:29:19.320864459 +0000 UTC m=+20.768247387"
Oct 31 00:29:20.112684 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:20.112650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:20.112841 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.112775    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:20.113140 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:20.113118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:20.113227 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.113205    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:20.214190 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:20.214152    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerStarted","Data":"0cd3cd9e725908620760504883198f0d58b1ab9f96bef5b70a5c348b5011e3d9"}
Oct 31 00:29:20.214831 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:20.214256    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:29:20.821294 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:20.821262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:20.821493 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.821433    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:20.821493 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.821493    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs podName:4d53981b-d854-4b47-b1f1-f46b0c810830 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:28.821475661 +0000 UTC m=+30.268858575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs") pod "network-metrics-daemon-2fk4z" (UID: "4d53981b-d854-4b47-b1f1-f46b0c810830") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:20.921596 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:20.921569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:20.921752 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:20.921605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:20.921752 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.921723    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:29:20.921752 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.921739    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:29:20.921752 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.921751    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bm7dt for pod openshift-network-diagnostics/network-check-target-mr44j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:20.921960 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.921796    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt podName:a5f64145-a6c9-4d5c-8951-348943d7806c nodeName:}" failed. No retries permitted until 2026-10-31 00:29:28.921783345 +0000 UTC m=+30.369166273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm7dt" (UniqueName: "kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt") pod "network-check-target-mr44j" (UID: "a5f64145-a6c9-4d5c-8951-348943d7806c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:20.921960 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.921876    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:20.921960 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:20.921901    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret podName:2a1760d4-777c-4853-bbbf-0a9afae8f45f nodeName:}" failed. No retries permitted until 2026-10-31 00:29:28.921892547 +0000 UTC m=+30.369275459 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret") pod "global-pull-secret-syncer-hrxqv" (UID: "2a1760d4-777c-4853-bbbf-0a9afae8f45f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:21.114644 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:21.114183    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:21.114644 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:21.114277    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:21.916607 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:21.916417    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:29:22.051708 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.051576    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:29:21.916604832Z","UUID":"fc13def6-be09-4959-affe-bf7c76ecf7d9","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:29:22.053067 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.053043    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:29:22.053199 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.053075    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:29:22.112305 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.112274    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:22.112469 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.112312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:22.112469 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:22.112425    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:22.112563 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:22.112515    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:22.221604 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.221567    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jh76v" event={"ID":"0aeb97e6-8dcd-4896-8a32-424681fcfc47","Type":"ContainerStarted","Data":"489f6a3d8e7b04c85f7916c3475b443b1fe951612369baddf45b6f815fcac7ef"}
Oct 31 00:29:22.224094 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.224068    2576 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="6f1d5fdc3d44f6548e93e74a0e6a76dd9940b7da650235db94b346bd8998bd07" exitCode=0
Oct 31 00:29:22.224218 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.224149    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerDied","Data":"6f1d5fdc3d44f6548e93e74a0e6a76dd9940b7da650235db94b346bd8998bd07"}
Oct 31 00:29:22.225973 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.225777    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wjjv6" event={"ID":"c36f1c1a-8072-409c-ba64-929fe596778b","Type":"ContainerStarted","Data":"b0c8159e0f4763add85522bfb75af3cd4d6b040a53aabcde035ecc857989a522"}
Oct 31 00:29:22.227716 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.227662    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerStarted","Data":"cd72a3ab4dddf0d39f55e1e4d976034e636b29ffabfdc2b8b08a2a328d093a24"}
Oct 31 00:29:22.337061 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.336953    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-jh76v" podStartSLOduration=1.633174243 podStartE2EDuration="9.336936273s" totalImagesPullingTime="7.70376203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:22.335959627 +0000 UTC m=+23.783342562" watchObservedRunningTime="2026-10-31 00:29:22.336936273 +0000 UTC m=+23.784319207"
Oct 31 00:29:22.463243 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:22.463181    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wjjv6" podStartSLOduration=2.474412401 podStartE2EDuration="10.463168488s" totalImagesPullingTime="7.988756087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:22.462937264 +0000 UTC m=+23.910320199" watchObservedRunningTime="2026-10-31 00:29:22.463168488 +0000 UTC m=+23.910551451"
Oct 31 00:29:23.057406 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:23.057325    2576 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" failed. No retries permitted until 2026-10-31 00:29:23.55729657 +0000 UTC m=+25.004679490 (durationBeforeRetry 500ms). Error: RegisterPlugin error -- plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: CSINode.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is invalid: metadata.ownerReferences.uid: Invalid value: "": must not be empty: rpc error: code = Unavailable desc = error reading from server: EOF
Oct 31 00:29:23.114224 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.114193    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:23.114390 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:23.114327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:23.232293 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.232248    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/0.log"
Oct 31 00:29:23.232746 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.232719    2576 generic.go:356] "Generic (PLEG): container finished" podID="3d584253-d14a-48e9-bfe6-563acb954e1f" containerID="cd72a3ab4dddf0d39f55e1e4d976034e636b29ffabfdc2b8b08a2a328d093a24" exitCode=1
Oct 31 00:29:23.233187 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.233166    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerDied","Data":"cd72a3ab4dddf0d39f55e1e4d976034e636b29ffabfdc2b8b08a2a328d093a24"}
Oct 31 00:29:23.260079 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.260052    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9cnsr"]
Oct 31 00:29:23.263285 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.263264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.314024 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.313948    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 00:29:23.314435 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.314412    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 00:29:23.314543 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.314491    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:29:23.314592 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.314552    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:29:23.354600 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.354545    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 00:29:23.354889 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.354838    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mqw2c\""
Oct 31 00:29:23.355002 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.354926    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 00:29:23.440944 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.440903    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-sys\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441115 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.440950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441115 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.441041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-tls\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441115 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.441099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-textfile\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441272 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.441125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pgxr\" (UniqueName: \"kubernetes.io/projected/02ae9325-803d-4acc-bafe-4349f1548558-kube-api-access-4pgxr\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441272 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.441150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-wtmp\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441272 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.441181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-metrics-client-ca\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441272 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.441215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-root\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.441272 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.441261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-root\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-sys\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-tls\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-textfile\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgxr\" (UniqueName: \"kubernetes.io/projected/02ae9325-803d-4acc-bafe-4349f1548558-kube-api-access-4pgxr\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-wtmp\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.544098 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.542330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-metrics-client-ca\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.545252 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.544811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-textfile\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.545252 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.544890    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-root\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.545252 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.544935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-sys\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.545252 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.544987    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-wtmp\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.545507 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.545389    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.545507 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.545443    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-metrics-client-ca\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.550371 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.548192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.550371 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.549003    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-tls\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.605872 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.605848    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgxr\" (UniqueName: \"kubernetes.io/projected/02ae9325-803d-4acc-bafe-4349f1548558-kube-api-access-4pgxr\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.880092 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:23.880026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:23.890659 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:23.890634    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ae9325_803d_4acc_bafe_4349f1548558.slice/crio-e6f65e92c3e360bce2705017fb50ebc484044bf53586582f1bb3596986e330c7 WatchSource:0}: Error finding container e6f65e92c3e360bce2705017fb50ebc484044bf53586582f1bb3596986e330c7: Status 404 returned error can't find the container with id e6f65e92c3e360bce2705017fb50ebc484044bf53586582f1bb3596986e330c7
Oct 31 00:29:24.111878 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.111846    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:24.112408 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:24.111967    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:24.112408 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.112045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:24.112408 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:24.112107    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:24.236451 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.236367    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2rxnb" event={"ID":"7c477db4-252e-4236-9ddd-79307115124f","Type":"ContainerStarted","Data":"64b504d4688aa3dcf34af37f605657b0d76e443fdb13c29a1f02af49214cd8f8"}
Oct 31 00:29:24.238429 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.238404    2576 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="4649aea2684dcd2226eb65d631da18548b2c7b246a1874f2bc6795a07a135e3c" exitCode=0
Oct 31 00:29:24.238549 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.238471    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerDied","Data":"4649aea2684dcd2226eb65d631da18548b2c7b246a1874f2bc6795a07a135e3c"}
Oct 31 00:29:24.240145 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.240114    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cntts" event={"ID":"eaaec7db-74f2-49ac-a56b-198f78680fce","Type":"ContainerStarted","Data":"a8f15d0c96af6ef2e1f80bb94288e8330b7c0110f426eaf378a2a3fa7486ed45"}
Oct 31 00:29:24.242864 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.242841    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/0.log"
Oct 31 00:29:24.248621 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.246917    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerStarted","Data":"7f7089a7649a1b2aae20648e94e9b99eda84513144df71bab6bb066d88b71070"}
Oct 31 00:29:24.248621 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.247439    2576 scope.go:122] "RemoveContainer" containerID="cd72a3ab4dddf0d39f55e1e4d976034e636b29ffabfdc2b8b08a2a328d093a24"
Oct 31 00:29:24.250115 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.250094    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cnsr" event={"ID":"02ae9325-803d-4acc-bafe-4349f1548558","Type":"ContainerStarted","Data":"e6f65e92c3e360bce2705017fb50ebc484044bf53586582f1bb3596986e330c7"}
Oct 31 00:29:24.351970 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.351930    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2rxnb" podStartSLOduration=1.357572311 podStartE2EDuration="11.351918172s" totalImagesPullingTime="9.994345861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:24.350642222 +0000 UTC m=+25.798025179" watchObservedRunningTime="2026-10-31 00:29:24.351918172 +0000 UTC m=+25.799301105"
Oct 31 00:29:24.357680 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.357656    2576 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:29:24.357975 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.357956    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:29:24.466542 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:24.466488    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cntts" podStartSLOduration=3.066950544 podStartE2EDuration="11.466468033s" totalImagesPullingTime="8.399517489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:24.465981468 +0000 UTC m=+25.913364424" watchObservedRunningTime="2026-10-31 00:29:24.466468033 +0000 UTC m=+25.913850968"
Oct 31 00:29:25.053171 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:25.053080    2576 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:29:25.053171 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:25.053012    2576 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"fc13def6-be09-4959-affe-bf7c76ecf7d9","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"}
Oct 31 00:29:25.114414 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:25.114388    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:25.114926 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:25.114503    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:25.255148 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:25.255118    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/0.log"
Oct 31 00:29:25.255670 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:25.255642    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerStarted","Data":"0e74c1c16c2a903f617696d0950c61d8572f4ed37fde6d1a9cf08c0059d66fbb"}
Oct 31 00:29:25.371296 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:25.371202    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" podStartSLOduration=3.5931847059999997 podStartE2EDuration="13.371190295s" totalImagesPullingTime="9.778005589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:25.370917472 +0000 UTC m=+26.818300408" watchObservedRunningTime="2026-10-31 00:29:25.371190295 +0000 UTC m=+26.818573228"
Oct 31 00:29:26.053842 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.053749    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:29:24.357974229Z","UUID":"c9264a07-e96a-454d-83ce-4e225fdda1cc","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:29:26.055122 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.055098    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:29:26.055236 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.055129    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:29:26.111705 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.111682    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:26.111842 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.111717    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:26.111842 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:26.111805    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:26.111949 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:26.111925    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:26.259606 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.259571    2576 generic.go:356] "Generic (PLEG): container finished" podID="02ae9325-803d-4acc-bafe-4349f1548558" containerID="15082ddd300228ad2b9b2c9c6720cb7a69eb89b3849ec6d7d31f73b1bf6fb097" exitCode=0
Oct 31 00:29:26.260085 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.259658    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cnsr" event={"ID":"02ae9325-803d-4acc-bafe-4349f1548558","Type":"ContainerDied","Data":"15082ddd300228ad2b9b2c9c6720cb7a69eb89b3849ec6d7d31f73b1bf6fb097"}
Oct 31 00:29:26.262329 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.262301    2576 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="c104edc53bd835027090be0e36b121eefc1210ffa6c966cea962eaca16b968b6" exitCode=0
Oct 31 00:29:26.262463 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:26.262384    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerDied","Data":"c104edc53bd835027090be0e36b121eefc1210ffa6c966cea962eaca16b968b6"}
Oct 31 00:29:27.067025 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:27.066801    2576 goroutinemap.go:150] Operation for "/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" failed. No retries permitted until 2026-10-31 00:29:27.566771371 +0000 UTC m=+29.014154474 (durationBeforeRetry 500ms). Error: RegisterPlugin error -- plugin registration failed with err: error updating CSINode object with CSI driver node info: error updating CSINode: timed out waiting for the condition; caused by: CSINode.storage.k8s.io "ip-10-0-131-97.us-west-2.compute.internal" is invalid: metadata.ownerReferences.uid: Invalid value: "": must not be empty: rpc error: code = Unavailable desc = error reading from server: EOF
Oct 31 00:29:27.112318 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.112293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:27.112462 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:27.112441    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:27.266645 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.266614    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:29:27.267507 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.267489    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/0.log"
Oct 31 00:29:27.267886 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.267866    2576 generic.go:356] "Generic (PLEG): container finished" podID="3d584253-d14a-48e9-bfe6-563acb954e1f" containerID="0e74c1c16c2a903f617696d0950c61d8572f4ed37fde6d1a9cf08c0059d66fbb" exitCode=1
Oct 31 00:29:27.267976 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.267955    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerDied","Data":"0e74c1c16c2a903f617696d0950c61d8572f4ed37fde6d1a9cf08c0059d66fbb"}
Oct 31 00:29:27.268049 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.268036    2576 scope.go:122] "RemoveContainer" containerID="cd72a3ab4dddf0d39f55e1e4d976034e636b29ffabfdc2b8b08a2a328d093a24"
Oct 31 00:29:27.268442 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.268427    2576 scope.go:122] "RemoveContainer" containerID="0e74c1c16c2a903f617696d0950c61d8572f4ed37fde6d1a9cf08c0059d66fbb"
Oct 31 00:29:27.268658 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:27.268633    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with CrashLoopBackOff: \"back-off 10s restarting failed container=csi-node-driver-registrar pod=aws-ebs-csi-driver-node-5jpqn_openshift-cluster-csi-drivers(3d584253-d14a-48e9-bfe6-563acb954e1f)\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" podUID="3d584253-d14a-48e9-bfe6-563acb954e1f"
Oct 31 00:29:27.270281 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.270262    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cnsr" event={"ID":"02ae9325-803d-4acc-bafe-4349f1548558","Type":"ContainerStarted","Data":"cbff99698243ae8ac5006114408d2538bdc5f7f67207c123d0001c1f1c092d97"}
Oct 31 00:29:27.270403 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.270301    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9cnsr" event={"ID":"02ae9325-803d-4acc-bafe-4349f1548558","Type":"ContainerStarted","Data":"fa081bf76c5ba79e9f8d803c7a2892d2c0e78b630d69c25c085cdf991ec85b3e"}
Oct 31 00:29:27.499849 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:27.499746    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9cnsr" podStartSLOduration=2.8094426070000003 podStartE2EDuration="4.499725594s" totalImagesPullingTime="1.605657513s" totalInitContainerRuntime="84.625474ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:27.499597824 +0000 UTC m=+28.946980770" watchObservedRunningTime="2026-10-31 00:29:27.499725594 +0000 UTC m=+28.947108555"
Oct 31 00:29:28.111957 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.111687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:28.111957 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.111694    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:28.112230 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.111960    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:28.112230 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.112043    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:28.278199 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.278127    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:29:28.282171 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.282139    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"0fd1207cf6f6996c21c7d250b157ad0e936ce9f16a7b8aafb9c772cf3c01a17a"}
Oct 31 00:29:28.282303 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.282251    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"f1dc34e51f65d3fb163a01c52b77da5cf6c2ba7331506dfd4192ea0e89df9e23"}
Oct 31 00:29:28.282303 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.282298    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"a915f9436ace4df640d73754e5c379f23c2bd60aeefd61bf6dba88517175e11b"}
Oct 31 00:29:28.282422 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.282340    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"484049f1744bc150bfe3cda055e04ed2bd27e7e83307a6e1b2709737a59d266d"}
Oct 31 00:29:28.282422 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.282411    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"b9479f35a1b5b03ddf86fdb6a04c82897f8e5506ca6ec9af048a85d65a0e3dc9"}
Oct 31 00:29:28.282515 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.282453    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"98b609ac5e41f37e8b8ba1923e48bb6dcf26e15dfd874443679de0ac61ceff08"}
Oct 31 00:29:28.887195 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.887166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:28.887380 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.887339    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:28.887440 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.887430    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs podName:4d53981b-d854-4b47-b1f1-f46b0c810830 nodeName:}" failed. No retries permitted until 2026-10-31 00:29:44.887414508 +0000 UTC m=+46.334797421 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs") pod "network-metrics-daemon-2fk4z" (UID: "4d53981b-d854-4b47-b1f1-f46b0c810830") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:29:28.989555 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.989528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:28.989733 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:28.989569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:28.989733 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.989664    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:28.989733 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.989681    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:29:28.989733 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.989700    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:29:28.989733 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.989712    2576 projected.go:196] Error preparing data for projected volume kube-api-access-bm7dt for pod openshift-network-diagnostics/network-check-target-mr44j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:28.989733 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.989728    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret podName:2a1760d4-777c-4853-bbbf-0a9afae8f45f nodeName:}" failed. No retries permitted until 2026-10-31 00:29:44.989713711 +0000 UTC m=+46.437096645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret") pod "global-pull-secret-syncer-hrxqv" (UID: "2a1760d4-777c-4853-bbbf-0a9afae8f45f") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:29:28.989924 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:28.989755    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt podName:a5f64145-a6c9-4d5c-8951-348943d7806c nodeName:}" failed. No retries permitted until 2026-10-31 00:29:44.989741611 +0000 UTC m=+46.437124541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bm7dt" (UniqueName: "kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt") pod "network-check-target-mr44j" (UID: "a5f64145-a6c9-4d5c-8951-348943d7806c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:29:29.113245 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:29.113179    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:29.113388 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:29.113261    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:29.287330 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:29.287300    2576 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="857cad3e8ce048b345cc95ba2617a455be93369d8b46182dbafe08527a1478d3" exitCode=0
Oct 31 00:29:29.287811 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:29.287338    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerDied","Data":"857cad3e8ce048b345cc95ba2617a455be93369d8b46182dbafe08527a1478d3"}
Oct 31 00:29:30.112079 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:30.112049    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:30.112338 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:30.112178    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:30.112338 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:30.112223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:30.112338 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:30.112293    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:30.292900 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:30.292871    2576 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="c82599d354f3cf9c16ba99048f723adc4c23211b17f0e2fabc7a4f01feadfec4" exitCode=0
Oct 31 00:29:30.293172 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:30.292921    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerDied","Data":"c82599d354f3cf9c16ba99048f723adc4c23211b17f0e2fabc7a4f01feadfec4"}
Oct 31 00:29:31.112166 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:31.112140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:31.112336 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:31.112238    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:31.298363 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:31.298312    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" event={"ID":"858636bd-15aa-47e2-95ce-daaf7cffb8d0","Type":"ContainerStarted","Data":"a9723c9f97ffc4cc78ca43b30655837026371f445c83943a248c85c223880d13"}
Oct 31 00:29:31.301341 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:31.301317    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"fe2d72542575ca19298a913eef555b308d59ef14814b25e096f254f8f19f0a46"}
Oct 31 00:29:31.423733 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:31.423637    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vx9mh" podStartSLOduration=5.613552128 podStartE2EDuration="18.423623836s" totalImagesPullingTime="12.810071708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:31.422779042 +0000 UTC m=+32.870161954" watchObservedRunningTime="2026-10-31 00:29:31.423623836 +0000 UTC m=+32.871006769"
Oct 31 00:29:32.112216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:32.112183    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:32.112216 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:32.112210    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:32.112519 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:32.112285    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:32.112519 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:32.112426    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:33.112754 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:33.112588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:33.113275 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:33.112832    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:33.310574 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:33.310544    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" event={"ID":"01df07af-c80e-4ae5-950d-87cc511cbda7","Type":"ContainerStarted","Data":"e7ed2fb94e85757b602c322e624433327b00dd3d54abe9235eded8a6c08a02f2"}
Oct 31 00:29:33.310785 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:33.310760    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:33.326055 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:33.326033    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:33.379288 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:33.379204    2576 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:33.379684 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:33.379667    2576 scope.go:122] "RemoveContainer" containerID="0e74c1c16c2a903f617696d0950c61d8572f4ed37fde6d1a9cf08c0059d66fbb"
Oct 31 00:29:33.379865 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:33.379843    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"csi-node-driver-registrar\" with CrashLoopBackOff: \"back-off 10s restarting failed container=csi-node-driver-registrar pod=aws-ebs-csi-driver-node-5jpqn_openshift-cluster-csi-drivers(3d584253-d14a-48e9-bfe6-563acb954e1f)\"" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" podUID="3d584253-d14a-48e9-bfe6-563acb954e1f"
Oct 31 00:29:33.502792 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:33.502731    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7" podStartSLOduration=7.147677552 podStartE2EDuration="21.50271451s" totalImagesPullingTime="14.355036958s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:33.484950789 +0000 UTC m=+34.932333723" watchObservedRunningTime="2026-10-31 00:29:33.50271451 +0000 UTC m=+34.950097444"
Oct 31 00:29:34.112366 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:34.112314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:34.112547 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:34.112375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:34.112547 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:34.112452    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:34.112658 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:34.112579    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:34.313196 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:34.313170    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:29:34.313753 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:34.313733    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:34.329018 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:34.328996    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:35.111997 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.111915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:35.112145 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:35.112068    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:35.176512 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.176469    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2fk4z"]
Oct 31 00:29:35.176695 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.176613    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:35.176760 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:35.176730    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:35.178757 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.178728    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mr44j"]
Oct 31 00:29:35.178879 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.178830    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:35.178956 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:35.178934    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:35.179309 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.179291    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxqv"]
Oct 31 00:29:35.315205 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.315177    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:35.315792 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:35.315231    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:29:35.315792 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:35.315272    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:36.317078 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:36.317045    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:29:37.111784 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:37.111759    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:37.111970 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:37.111792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:37.111970 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:37.111765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:37.111970 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:37.111844    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:37.111970 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:37.111931    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:37.112180 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:37.112029    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:39.113055 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.112885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:39.113583 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:39.113114    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2fk4z" podUID="4d53981b-d854-4b47-b1f1-f46b0c810830"
Oct 31 00:29:39.113583 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.113001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:39.113583 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.112975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:39.113583 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:39.113463    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mr44j" podUID="a5f64145-a6c9-4d5c-8951-348943d7806c"
Oct 31 00:29:39.113583 ip-10-0-131-97 kubenswrapper[2576]: E1031 00:29:39.113538    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hrxqv" podUID="2a1760d4-777c-4853-bbbf-0a9afae8f45f"
Oct 31 00:29:39.113826 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.113604    2576 scope.go:122] "RemoveContainer" containerID="0e74c1c16c2a903f617696d0950c61d8572f4ed37fde6d1a9cf08c0059d66fbb"
Oct 31 00:29:39.238330 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.238293    2576 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:29:39.238888 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.238869    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:29:39.325210 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.325183    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:29:39.325795 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:39.325769    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn" event={"ID":"3d584253-d14a-48e9-bfe6-563acb954e1f","Type":"ContainerStarted","Data":"aed03e8c37cd99f69e88f168b51e685db1ccf13ec322aa2239a6f28a51deb217"}
Oct 31 00:29:40.062343 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.062282    2576 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"c9264a07-e96a-454d-83ce-4e225fdda1cc","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"}
Oct 31 00:29:40.062511 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.062368    2576 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:29:40.665562 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.665533    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeReady"
Oct 31 00:29:40.665927 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.665659    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 00:29:40.820466 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.820397    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-knvdm"]
Oct 31 00:29:40.825767 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.825242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:40.827225 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.827196    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nm5ww"]
Oct 31 00:29:40.833618 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.833602    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kdkdw"]
Oct 31 00:29:40.833729 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.833713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.843215 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.843197    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.872691 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-config-volume\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.872691 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-tmp-dir\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.872881 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlj5w\" (UniqueName: \"kubernetes.io/projected/055635e6-bb27-4685-8f8e-f26d873567a8-kube-api-access-mlj5w\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:40.872881 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055635e6-bb27-4685-8f8e-f26d873567a8-cert\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:40.872881 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454cfe49-4029-4326-b0ce-f3fe9194d991-crio-socket\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.872881 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872854    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hqnn\" (UniqueName: \"kubernetes.io/projected/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-api-access-2hqnn\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.873083 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.873083 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454cfe49-4029-4326-b0ce-f3fe9194d991-data-volume\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.873083 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-metrics-tls\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.873083 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.872996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb98m\" (UniqueName: \"kubernetes.io/projected/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-kube-api-access-bb98m\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.873083 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.873010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454cfe49-4029-4326-b0ce-f3fe9194d991-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.888496 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.888474    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:29:40.888496 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.888475    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:29:40.888651 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.888562    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:29:40.888651 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.888568    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:29:40.888651 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.888575    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:29:40.890223 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.890207    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:29:40.890317 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.890282    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:29:40.890382 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.890369    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bpzsc\""
Oct 31 00:29:40.890414 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.890401    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wzhm5\""
Oct 31 00:29:40.896560 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.896540    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vb9k\""
Oct 31 00:29:40.901560 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.901539    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-knvdm"]
Oct 31 00:29:40.939367 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.939321    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nm5ww"]
Oct 31 00:29:40.968138 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.968107    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kdkdw"]
Oct 31 00:29:40.974090 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-config-volume\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.974090 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-tmp-dir\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.974277 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974119    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlj5w\" (UniqueName: \"kubernetes.io/projected/055635e6-bb27-4685-8f8e-f26d873567a8-kube-api-access-mlj5w\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:40.974277 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055635e6-bb27-4685-8f8e-f26d873567a8-cert\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:40.974277 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454cfe49-4029-4326-b0ce-f3fe9194d991-crio-socket\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.974277 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqnn\" (UniqueName: \"kubernetes.io/projected/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-api-access-2hqnn\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.974277 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974212    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.974517 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454cfe49-4029-4326-b0ce-f3fe9194d991-data-volume\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.974517 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454cfe49-4029-4326-b0ce-f3fe9194d991-crio-socket\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.974517 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-metrics-tls\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.974517 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974466    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-tmp-dir\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.974517 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974477    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb98m\" (UniqueName: \"kubernetes.io/projected/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-kube-api-access-bb98m\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.974703 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454cfe49-4029-4326-b0ce-f3fe9194d991-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.974703 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.974693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454cfe49-4029-4326-b0ce-f3fe9194d991-data-volume\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.975171 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.975148    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.977989 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.977965    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454cfe49-4029-4326-b0ce-f3fe9194d991-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:40.978082 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.978005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055635e6-bb27-4685-8f8e-f26d873567a8-cert\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:40.986646 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.986630    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:29:40.990141 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.990122    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:29:40.994604 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.994585    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-config-volume\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:40.996929 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:40.996910    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-metrics-tls\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:41.034960 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.034935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlj5w\" (UniqueName: \"kubernetes.io/projected/055635e6-bb27-4685-8f8e-f26d873567a8-kube-api-access-mlj5w\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:41.034960 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.034945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb98m\" (UniqueName: \"kubernetes.io/projected/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-kube-api-access-bb98m\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:41.036791 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.036771    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqnn\" (UniqueName: \"kubernetes.io/projected/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-api-access-2hqnn\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:41.062636 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.062540    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:29:39.238884527Z","UUID":"99242f3a-7f05-4feb-90e5-a9919858d7f5","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:29:41.063777 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.063748    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:29:41.063777 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.063778    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:29:41.112828 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.112743    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:41.112828 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.112783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:41.113064 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.112750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:41.141278 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.141249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:41.157208 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.157189    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:41.165401 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.165379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:41.169482 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.169463    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:29:41.169652 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.169625    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6z87r\""
Oct 31 00:29:41.169652 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.169625    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:29:41.169784 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.169650    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9b4md\""
Oct 31 00:29:41.169784 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.169688    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-prxzz\""
Oct 31 00:29:41.169895 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.169806    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:29:41.169895 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.169839    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:29:41.647842 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.647564    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kdkdw"]
Oct 31 00:29:41.651302 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.651278    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-knvdm"]
Oct 31 00:29:41.652194 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:41.652172    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nm5ww"]
Oct 31 00:29:41.654112 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:41.654088    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055635e6_bb27_4685_8f8e_f26d873567a8.slice/crio-4b3f141556a418cea1aab93857408463bb46f735d0c768878b7f321ff25a36d5 WatchSource:0}: Error finding container 4b3f141556a418cea1aab93857408463bb46f735d0c768878b7f321ff25a36d5: Status 404 returned error can't find the container with id 4b3f141556a418cea1aab93857408463bb46f735d0c768878b7f321ff25a36d5
Oct 31 00:29:41.668415 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:41.668387    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod454cfe49_4029_4326_b0ce_f3fe9194d991.slice/crio-7b6e21ce95b1d765a559ae1d4cccdf70c0f5e8451b3bac547fab779819a14737 WatchSource:0}: Error finding container 7b6e21ce95b1d765a559ae1d4cccdf70c0f5e8451b3bac547fab779819a14737: Status 404 returned error can't find the container with id 7b6e21ce95b1d765a559ae1d4cccdf70c0f5e8451b3bac547fab779819a14737
Oct 31 00:29:42.333208 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:42.333175    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdkdw" event={"ID":"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf","Type":"ContainerStarted","Data":"2a2d8ef9d79978edb55e4c9a03cc42df2ec6e59d5ed7176f67d0cc2199c7fd2e"}
Oct 31 00:29:42.334283 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:42.334259    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-knvdm" event={"ID":"055635e6-bb27-4685-8f8e-f26d873567a8","Type":"ContainerStarted","Data":"4b3f141556a418cea1aab93857408463bb46f735d0c768878b7f321ff25a36d5"}
Oct 31 00:29:42.335626 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:42.335598    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5ww" event={"ID":"454cfe49-4029-4326-b0ce-f3fe9194d991","Type":"ContainerStarted","Data":"229f2e6ada1da9fd19c3ddbc696aca23964a5a637d1f421e0d4fa8e9e75802ab"}
Oct 31 00:29:42.335709 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:42.335639    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5ww" event={"ID":"454cfe49-4029-4326-b0ce-f3fe9194d991","Type":"ContainerStarted","Data":"7b6e21ce95b1d765a559ae1d4cccdf70c0f5e8451b3bac547fab779819a14737"}
Oct 31 00:29:43.062532 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:43.062499    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:43.063184 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:43.062656    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:29:43.063184 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:43.063066    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:43.250729 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:43.250699    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:43.250932 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:43.250919    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:29:43.268656 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:43.268626    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:44.343509 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:44.343478    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5ww" event={"ID":"454cfe49-4029-4326-b0ce-f3fe9194d991","Type":"ContainerStarted","Data":"745de6aa6ca6abfa7db553ab5d8a8b1bf1df73db989b3717f65435f3657d8a8d"}
Oct 31 00:29:44.909274 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:44.909236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:44.912976 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:44.912951    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:45.010955 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.010453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:45.010955 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.010503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:45.013659 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.013626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:45.014969 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.014940    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:45.047314 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.047280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:45.069500 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.069469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:45.113427 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.113243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:45.349564 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.349528    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdkdw" event={"ID":"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf","Type":"ContainerStarted","Data":"7dccd3edf2733258dd2d2ea8147a06d64f56eb24d2908d734a407dca3c364cc6"}
Oct 31 00:29:45.474955 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.474891    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hrxqv"]
Oct 31 00:29:45.483297 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:45.483265    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1760d4_777c_4853_bbbf_0a9afae8f45f.slice/crio-11eb5d7869d8477964c2140614c7d3f0c2f9c2ee4f8f19b690c7f218c68e2fba WatchSource:0}: Error finding container 11eb5d7869d8477964c2140614c7d3f0c2f9c2ee4f8f19b690c7f218c68e2fba: Status 404 returned error can't find the container with id 11eb5d7869d8477964c2140614c7d3f0c2f9c2ee4f8f19b690c7f218c68e2fba
Oct 31 00:29:45.522973 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.522904    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mr44j"]
Oct 31 00:29:45.528287 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:45.528251    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f64145_a6c9_4d5c_8951_348943d7806c.slice/crio-af99bd996b2d98a9d885edc409f4aa9dab998e5c29f615601fafce63b12dcb72 WatchSource:0}: Error finding container af99bd996b2d98a9d885edc409f4aa9dab998e5c29f615601fafce63b12dcb72: Status 404 returned error can't find the container with id af99bd996b2d98a9d885edc409f4aa9dab998e5c29f615601fafce63b12dcb72
Oct 31 00:29:45.590925 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:45.590896    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2fk4z"]
Oct 31 00:29:45.595577 ip-10-0-131-97 kubenswrapper[2576]: W1031 00:29:45.595546    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d53981b_d854_4b47_b1f1_f46b0c810830.slice/crio-fee5b4908ed32bb9d799e56f477e2ae124cfcd74d46963364e061521c66025d9 WatchSource:0}: Error finding container fee5b4908ed32bb9d799e56f477e2ae124cfcd74d46963364e061521c66025d9: Status 404 returned error can't find the container with id fee5b4908ed32bb9d799e56f477e2ae124cfcd74d46963364e061521c66025d9
Oct 31 00:29:46.355007 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:46.354973    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kdkdw" event={"ID":"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf","Type":"ContainerStarted","Data":"0e2ebfbf45f2eec906ac4f9b5f6a3dbc5f646106a3d45d8530f97cb2670ebea6"}
Oct 31 00:29:46.356459 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:46.355057    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:46.358275 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:46.358247    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fk4z" event={"ID":"4d53981b-d854-4b47-b1f1-f46b0c810830","Type":"ContainerStarted","Data":"fee5b4908ed32bb9d799e56f477e2ae124cfcd74d46963364e061521c66025d9"}
Oct 31 00:29:46.362783 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:46.362746    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hrxqv" event={"ID":"2a1760d4-777c-4853-bbbf-0a9afae8f45f","Type":"ContainerStarted","Data":"11eb5d7869d8477964c2140614c7d3f0c2f9c2ee4f8f19b690c7f218c68e2fba"}
Oct 31 00:29:46.364036 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:46.364012    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mr44j" event={"ID":"a5f64145-a6c9-4d5c-8951-348943d7806c","Type":"ContainerStarted","Data":"af99bd996b2d98a9d885edc409f4aa9dab998e5c29f615601fafce63b12dcb72"}
Oct 31 00:29:46.511472 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:46.511375    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-kdkdw" podStartSLOduration=3.060729824 podStartE2EDuration="6.511340858s" totalImagesPullingTime="3.450611034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:46.510234458 +0000 UTC m=+47.957617406" watchObservedRunningTime="2026-10-31 00:29:46.511340858 +0000 UTC m=+47.958723789"
Oct 31 00:29:47.370809 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:47.370384    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-knvdm" event={"ID":"055635e6-bb27-4685-8f8e-f26d873567a8","Type":"ContainerStarted","Data":"402f63e43962bbe4eb725b5f2db2ba74e318819f5b9e5fd4791a2b9d44817f4f"}
Oct 31 00:29:47.373185 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:47.373152    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nm5ww" event={"ID":"454cfe49-4029-4326-b0ce-f3fe9194d991","Type":"ContainerStarted","Data":"13998d980dff77922edec860d41973130d04099dc3067eab87a24d6dfd9a317e"}
Oct 31 00:29:47.484172 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:47.484101    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-knvdm" podStartSLOduration=2.266934389 podStartE2EDuration="7.484081842s" totalImagesPullingTime="5.217147453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:47.48298983 +0000 UTC m=+48.930372763" watchObservedRunningTime="2026-10-31 00:29:47.484081842 +0000 UTC m=+48.931464780"
Oct 31 00:29:47.599602 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:47.599550    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nm5ww" podStartSLOduration=3.163462015 podStartE2EDuration="7.599535301s" totalImagesPullingTime="4.436073286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:47.59851386 +0000 UTC m=+49.045896800" watchObservedRunningTime="2026-10-31 00:29:47.599535301 +0000 UTC m=+49.046918233"
Oct 31 00:29:48.381286 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:48.380657    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fk4z" event={"ID":"4d53981b-d854-4b47-b1f1-f46b0c810830","Type":"ContainerStarted","Data":"426f459fca2791c6060672b4791fd92fdfc7321271b83e43af44c10106c53e31"}
Oct 31 00:29:48.381286 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:48.380711    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2fk4z" event={"ID":"4d53981b-d854-4b47-b1f1-f46b0c810830","Type":"ContainerStarted","Data":"72a14b6721c8288d5529cd790fceecc51d538fa5ced3856db4d26901b585a92e"}
Oct 31 00:29:48.493745 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:48.493678    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2fk4z" podStartSLOduration=32.989111786 podStartE2EDuration="35.493659317s" totalImagesPullingTime="2.504547531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:48.492976842 +0000 UTC m=+49.940359776" watchObservedRunningTime="2026-10-31 00:29:48.493659317 +0000 UTC m=+49.941042251"
Oct 31 00:29:50.388166 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:50.388071    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mr44j" event={"ID":"a5f64145-a6c9-4d5c-8951-348943d7806c","Type":"ContainerStarted","Data":"87c5bd6afd50c9665d938ea5cbfdd3847b76ec5a4f53114101a17755a07907de"}
Oct 31 00:29:50.388647 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:50.388447    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:50.502051 ip-10-0-131-97 kubenswrapper[2576]: I1031 00:29:50.501989    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mr44j" podStartSLOduration=32.958839236 podStartE2EDuration="37.501975476s" totalImagesPullingTime="4.54313624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:50.501625266 +0000 UTC m=+51.949008202" watchObservedRunningTime="2026-10-31 00:29:50.501975476 +0000 UTC m=+51.949358439"
Oct 31 00:29:51.389251 ip-10-0-131-97 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 00:29:51.401830 ip-10-0-131-97 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 00:29:51.401996 ip-10-0-131-97 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 00:29:51.402154 ip-10-0-131-97 systemd[1]: kubelet.service: Consumed 2.882s CPU time.
Oct 31 00:29:51.415038 ip-10-0-131-97 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:29:51.473180 ip-10-0-131-97 kubenswrapper[4888]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:29:51.473180 ip-10-0-131-97 kubenswrapper[4888]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:29:51.473180 ip-10-0-131-97 kubenswrapper[4888]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:29:51.473180 ip-10-0-131-97 kubenswrapper[4888]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:29:51.475164 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475092    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.475164 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475164    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.475164 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475167    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475170    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475173    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475176    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475178    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475181    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475183    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475185    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475188    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475190    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475193    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475196    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475199    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475201    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475204    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475206    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475209    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475212    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475214    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.475256 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475217    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475219    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475222    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475225    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475229    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475232    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475239    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475241    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475244    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475247    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475251    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475253    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475256    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475258    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475261    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475264    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475266    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475269    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475271    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.475856 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475276    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475280    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475283    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475286    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475288    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475291    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475293    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475296    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475298    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475301    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475303    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475305    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475308    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475310    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475312    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475315    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475317    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475319    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475322    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475324    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.476323 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475328    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475331    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475333    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475336    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475340    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475342    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475358    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475361    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475363    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475366    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475368    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475371    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475373    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475376    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475378    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475381    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475383    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475385    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475388    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.476845 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475391    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475393    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475396    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475398    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475401    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475403    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475406    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475408    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475411    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475413    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475416    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475418    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475421    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475423    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475427    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475430    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475432    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475435    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475437    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475440    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.477320 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475442    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475445    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475447    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475450    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475452    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475456    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.475458    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476160    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476165    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476168    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476171    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476173    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476176    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476179    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476182    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476184    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476186    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476189    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476192    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476194    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.477828 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476197    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476199    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476202    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476204    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476207    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476211    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476214    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476218    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476222    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476225    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476227    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476230    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476232    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476235    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476237    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476239    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476242    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476244    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476246    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.478343 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476249    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476251    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476254    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476256    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476258    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476261    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476264    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476266    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476269    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476271    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476273    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476276    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476278    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476280    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476283    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476285    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476287    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476289    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476292    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476294    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.478813 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476297    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476300    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476304    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476308    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476310    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476313    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476315    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476318    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476320    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476322    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476325    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476327    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476330    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476333    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476335    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476338    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476340    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476343    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476360    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.479763 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476363    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476365    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476368    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476377    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476380    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476382    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476385    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476387    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476389    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476392    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476395    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476397    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476399    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476402    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476405    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476408    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476411    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476413    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476416    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476418    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.480221 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476420    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476423    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476426    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476429    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476431    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476434    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476436    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476439    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476441    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476444    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476446    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476449    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476451    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476453    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476456    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476911    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476915    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476918    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476921    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476923    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.480732 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476926    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476928    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476931    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476933    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476936    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476939    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476941    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476944    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476947    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476949    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476952    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476954    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476957    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476959    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476962    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476964    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476967    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476969    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476972    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476974    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.481209 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476976    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476979    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476981    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476983    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476986    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476988    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476990    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476993    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476995    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.476997    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477000    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477002    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477008    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477011    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477013    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477016    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477019    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477022    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477024    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477027    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.481696 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477030    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477032    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477035    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477037    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477040    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477042    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477045    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477047    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477049    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477051    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477054    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477057    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477059    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477062    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477065    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477067    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477069    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477072    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477074    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477076    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.482159 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477079    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477081    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477085    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477088    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477092    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477095    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477097    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477100    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477103    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477105    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477108    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477110    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477112    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477115    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477118    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477120    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477123    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477125    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477127    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.482665 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477129    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477132    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477134    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477136    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477139    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477141    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477143    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477146    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477148    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477150    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477153    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477155    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477157    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477160    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477162    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477165    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477169    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477173    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477176    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.483217 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477180    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477183    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477186    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477275    4888 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477282    4888 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477287    4888 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477291    4888 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477297    4888 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477302    4888 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477308    4888 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477314    4888 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477319    4888 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477323    4888 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477327    4888 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477331    4888 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477335    4888 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477338    4888 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477340    4888 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477343    4888 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477358    4888 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477361    4888 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477365    4888 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477368    4888 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:29:51.483971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477371    4888 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477373    4888 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477377    4888 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477381    4888 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477383    4888 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477386    4888 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477391    4888 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477395    4888 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477398    4888 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477401    4888 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477404    4888 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477406    4888 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477410    4888 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477413    4888 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477416    4888 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477418    4888 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477421    4888 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477424    4888 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477428    4888 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477431    4888 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477433    4888 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477437    4888 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477439    4888 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477443    4888 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:29:51.484689 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477446    4888 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477449    4888 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477452    4888 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477455    4888 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477458    4888 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477460    4888 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477463    4888 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477466    4888 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477469    4888 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477471    4888 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477475    4888 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477478    4888 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477481    4888 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477484    4888 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477488    4888 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477492    4888 flags.go:64] FLAG: --help="false"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477495    4888 flags.go:64] FLAG: --hostname-override="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477498    4888 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477501    4888 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477504    4888 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477507    4888 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477511    4888 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477513    4888 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:29:51.485273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477516    4888 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477519    4888 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477521    4888 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477524    4888 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477528    4888 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477531    4888 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477534    4888 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477536    4888 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477539    4888 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477542    4888 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477544    4888 flags.go:64] FLAG: --lock-file=""
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477547    4888 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477549    4888 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477552    4888 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477557    4888 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477560    4888 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477562    4888 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477565    4888 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477568    4888 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477571    4888 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477573    4888 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477576    4888 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477580    4888 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477584    4888 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477589    4888 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:29:51.485836 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477592    4888 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477595    4888 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477597    4888 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477600    4888 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477603    4888 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477606    4888 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477609    4888 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477615    4888 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477618    4888 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477621    4888 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477623    4888 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477626    4888 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477629    4888 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477632    4888 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477635    4888 flags.go:64] FLAG: --port="10250"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477638    4888 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477640    4888 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0daf3ae3370a4ac65"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477643    4888 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477646    4888 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477649    4888 flags.go:64] FLAG: --register-node="true"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477651    4888 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477655    4888 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477657    4888 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477660    4888 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477662    4888 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477666    4888 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:29:51.486418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477669    4888 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477671    4888 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477674    4888 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477677    4888 flags.go:64] FLAG: --runonce="false"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477681    4888 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477685    4888 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477688    4888 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477690    4888 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477693    4888 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477696    4888 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477699    4888 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477701    4888 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477704    4888 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477707    4888 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477709    4888 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477712    4888 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477715    4888 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477717    4888 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477720    4888 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477725    4888 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477728    4888 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477731    4888 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477739    4888 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477742    4888 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477745    4888 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:29:51.487054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477747    4888 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477750    4888 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477753    4888 flags.go:64] FLAG: --v="2"
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477757    4888 flags.go:64] FLAG: --version="false"
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477761    4888 flags.go:64] FLAG: --vmodule=""
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477764    4888 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.477767    4888 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477888    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477891    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477894    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477897    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477901    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477905    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477907    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477910    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477912    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477915    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477917    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477919    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477922    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477924    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.487654 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477927    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477929    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477931    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477934    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477936    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477938    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477941    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477944    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477946    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477948    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477951    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477953    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477956    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477958    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477960    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477963    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477965    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477967    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477970    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477972    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.488219 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477974    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477977    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477980    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477985    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477987    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477990    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477992    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477995    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477997    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.477999    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478002    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478004    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478006    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478009    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478011    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478013    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478016    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478018    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478020    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478024    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.488805 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478027    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478029    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478031    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478034    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478036    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478038    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478041    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478044    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478046    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478048    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478052    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478056    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478059    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478061    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478065    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478068    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478071    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478073    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478076    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.489317 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478078    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478080    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478083    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478085    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478088    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478090    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478093    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478096    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478099    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478102    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478104    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478106    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478109    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478111    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478114    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478117    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478119    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478122    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478124    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.489892 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478126    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478129    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478131    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478133    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478136    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478138    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478140    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478143    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478146    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478150    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478152    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478155    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478157    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478160    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.478184    4888 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478221    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478226    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.490358 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478229    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478231    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478234    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478236    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478239    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478241    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478244    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478246    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478249    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478252    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478256    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478258    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478261    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478263    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478267    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478271    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478274    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478277    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478279    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.490897 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478281    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478284    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478286    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478288    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478292    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478294    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478298    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478300    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478303    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478305    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478308    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478312    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478315    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478318    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478320    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478323    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478325    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478328    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478330    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.491520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478332    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478335    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478337    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478339    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478342    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478357    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478360    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478363    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478365    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478368    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478370    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478373    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478375    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478377    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478379    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478382    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478384    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478388    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478390    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478394    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.491998 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478396    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478399    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478401    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478403    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478406    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478408    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478411    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478413    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478415    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478418    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478420    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478423    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478425    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478427    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478430    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478432    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478434    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478437    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478440    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.492514 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478442    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478444    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478447    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478449    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478451    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478454    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478456    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478458    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478461    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478463    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478466    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478469    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478472    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478475    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478477    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478479    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478481    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478484    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478486    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478488    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.492959 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478491    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478493    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478496    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478498    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478500    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478502    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.478506    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.482446    4888 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.482460    4888 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.482477    4888 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.482482    4888 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482553    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482558    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482561    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482564    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482567    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482569    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482572    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482574    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.493450 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482577    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482580    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482582    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482585    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482587    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482589    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482592    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482595    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482597    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482600    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482603    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482605    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482607    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482610    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482612    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482615    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482617    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482620    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482622    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482625    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.493902 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482628    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482630    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482633    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482635    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482638    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482645    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482649    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482653    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482656    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482659    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482661    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482664    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482666    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482669    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482672    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482674    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482677    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482680    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482682    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.494387 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482684    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482687    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482690    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482692    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482695    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482697    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482700    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482702    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482705    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482713    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482716    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482718    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482721    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482723    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482726    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482728    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482731    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482733    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482736    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482743    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.494846 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482746    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482748    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482751    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482760    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482764    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482766    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482769    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482771    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482773    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482776    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482778    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482781    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482783    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482785    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482788    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482790    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482793    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482796    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482798    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482800    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.495315 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482803    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482805    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482807    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482810    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482813    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482815    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482817    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482821    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482825    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482828    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482830    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482833    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482842    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482844    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482847    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482849    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482851    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482853    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.495793 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.482856    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.482882    4888 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483880    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483894    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483899    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483908    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483914    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483919    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483924    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483929    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483933    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483937    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483941    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483945    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483949    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.496214 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483953    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483957    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483965    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483970    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483974    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483978    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483982    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483987    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483991    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483995    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.483999    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484003    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484038    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484042    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484046    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484055    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484058    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484062    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484066    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.496585 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484070    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484074    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484077    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484081    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484087    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484093    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484098    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484103    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484112    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484116    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484120    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484124    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484128    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484132    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484136    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484140    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484144    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484150    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484153    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.497057 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484156    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484159    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484161    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484164    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484166    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484169    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484172    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484185    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484189    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484193    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484196    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484205    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484209    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484213    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484217    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484221    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484225    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484230    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484233    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.497533 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484237    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484241    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484245    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484249    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484257    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484261    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484266    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484270    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484273    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484278    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484282    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484286    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484289    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484294    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484298    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484303    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484306    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484311    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484314    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484316    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.497975 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484338    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484383    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484387    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484482    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484488    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484491    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484493    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484496    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484498    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484500    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484503    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484507    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484509    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484512    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484514    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484516    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.498556 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.484549    4888 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484589    4888 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484593    4888 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484597    4888 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484600    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484624    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484628    4888 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484631    4888 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484633    4888 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484636    4888 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484638    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484641    4888 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484643    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484646    4888 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484648    4888 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484650    4888 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484653    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484655    4888 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484658    4888 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484661    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:29:51.498942 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484663    4888 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484666    4888 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484668    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484670    4888 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484673    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484675    4888 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484678    4888 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484680    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484683    4888 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484687    4888 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484690    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484693    4888 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484696    4888 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484698    4888 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484701    4888 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484704    4888 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484707    4888 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484709    4888 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484712    4888 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:29:51.499417 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484714    4888 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484717    4888 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484719    4888 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484722    4888 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484724    4888 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484728    4888 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484731    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484733    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484735    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484738    4888 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484740    4888 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484742    4888 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484745    4888 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484748    4888 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484750    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484752    4888 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484755    4888 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484757    4888 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484762    4888 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:29:51.499868 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484765    4888 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484767    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484771    4888 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484775    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484778    4888 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484781    4888 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484783    4888 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484786    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484788    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484791    4888 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484794    4888 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484797    4888 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484799    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484802    4888 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484804    4888 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484807    4888 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484809    4888 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484811    4888 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484815    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:29:51.500303 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484817    4888 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484820    4888 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484823    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484825    4888 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484827    4888 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484829    4888 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484832    4888 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484834    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484836    4888 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484839    4888 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484841    4888 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484845    4888 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484849    4888 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484852    4888 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484854    4888 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484856    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484859    4888 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484862    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484864    4888 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484866    4888 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:29:51.500754 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484869    4888 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484872    4888 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484874    4888 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484877    4888 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484880    4888 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484882    4888 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484885    4888 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484887    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484889    4888 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:29:51.484892    4888 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.484949    4888 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.485931    4888 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.485981    4888 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.486186    4888 server.go:1014] "Starting client certificate rotation"
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.486319    4888 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.486379    4888 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 00:24:07 +0000 UTC" deadline="2028-07-15 04:14:04.897357284 +0000 UTC"
Oct 31 00:29:51.501401 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.486405    4888 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14955h44m13.410954137s"
Oct 31 00:29:51.501999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.486625    4888 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:29:51.501999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.487686    4888 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:29:51.501999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.489059    4888 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:29:51.501999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.490568    4888 log.go:25] "Validated CRI v1 image API"
Oct 31 00:29:51.501999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.491413    4888 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:29:51.501999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.493159    4888 fs.go:135] Filesystem UUIDs: map[41126c8c-219f-4105-8c86-6be80465a1a4:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8d53334c-f11a-4089-9695-4bdf1e2b2636:/dev/nvme0n1p4]
Oct 31 00:29:51.502227 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.493179    4888 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06a421cfd426cf060983d93846152acb56e9cb14ba8f96e860b5cbb13ce391ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06a421cfd426cf060983d93846152acb56e9cb14ba8f96e860b5cbb13ce391ec/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/11eb5d7869d8477964c2140614c7d3f0c2f9c2ee4f8f19b690c7f218c68e2fba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/11eb5d7869d8477964c2140614c7d3f0c2f9c2ee4f8f19b690c7f218c68e2fba/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1223b71cf0a603c934d3f9c07558f7c8a570f784f970a554124ed3bc3020caae/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1223b71cf0a603c934d3f9c07558f7c8a570f784f970a554124ed3bc3020caae/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a2d8ef9d79978edb55e4c9a03cc42df2ec6e59d5ed7176f67d0cc2199c7fd2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a2d8ef9d79978edb55e4c9a03cc42df2ec6e59d5ed7176f67d0cc2199c7fd2e/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32adff65999d9cdc6af88443bc0074345790dc83bd70c2344f52fe4279ceff50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32adff65999d9cdc6af88443bc0074345790dc83bd70c2344f52fe4279ceff50/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4b3f141556a418cea1aab93857408463bb46f735d0c768878b7f321ff25a36d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4b3f141556a418cea1aab93857408463bb46f735d0c768878b7f321ff25a36d5/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f898bf492e7ce0e4cd1003a6c9536d8f2f903bc7d8d29cc122aa5b41dcefe8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f898bf492e7ce0e4cd1003a6c9536d8f2f903bc7d8d29cc122aa5b41dcefe8e/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6843b645ac49edfc71489a9ed5110c48402c455d35ebb81e86185affa9e03652/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6843b645ac49edfc71489a9ed5110c48402c455d35ebb81e86185affa9e03652/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b6e21ce95b1d765a559ae1d4cccdf70c0f5e8451b3bac547fab779819a14737/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b6e21ce95b1d765a559ae1d4cccdf70c0f5e8451b3bac547fab779819a14737/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87caaa89b01db633825a45da0a845742cec1b15a776d8180df82f7dc6e9fc728/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87caaa89b01db633825a45da0a845742cec1b15a776d8180df82f7dc6e9fc728/userdata/shm major:0 minor:108 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8d2e85eca66a1429f17d4cbc5e4f7613bf3e15d0017439c4a2a7991ef0bdd08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8d2e85eca66a1429f17d4cbc5e4f7613bf3e15d0017439c4a2a7991ef0bdd08/userdata/shm major:0 minor:102 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af99bd996b2d98a9d885edc409f4aa9dab998e5c29f615601fafce63b12dcb72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af99bd996b2d98a9d885edc409f4aa9dab998e5c29f615601fafce63b12dcb72/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d392df2bf42859cf3facac13cbe5f1b870941235d40ddb1482e793fb18b275d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d392df2bf42859cf3facac13cbe5f1b870941235d40ddb1482e793fb18b275d9/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6f65e92c3e360bce2705017fb50ebc484044bf53586582f1bb3596986e330c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6f65e92c3e360bce2705017fb50ebc484044bf53586582f1bb3596986e330c7/userdata/shm major:0 minor:142 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fee5b4908ed32bb9d799e56f477e2ae124cfcd74d46963364e061521c66025d9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fee5b4908ed32bb9d799e56f477e2ae124cfcd74d46963364e061521c66025d9/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volumes/kubernetes.io~projected/kube-api-access-bm2hb:{mountpoint:/var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volumes/kubernetes.io~projected/kube-api-access-bm2hb major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~projected/kube-api-access-4pgxr:{mountpoint:/var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~projected/kube-api-access-4pgxr major:0 minor:151 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:147 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:150 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/055635e6-bb27-4685-8f8e-f26d873567a8/volumes/kubernetes.io~projected/kube-api-access-mlj5w:{mountpoint:/var/lib/kubelet/pods/055635e6-bb27-4685-8f8e-f26d873567a8/volumes/kubernetes.io~projected/kube-api-access-mlj5w major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/055635e6-bb27-4685-8f8e-f26d873567a8/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/055635e6-bb27-4685-8f8e-f26d873567a8/volumes/kubernetes.io~secret/cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0aeb97e6-8dcd-4896-8a32-424681fcfc47/volumes/kubernetes.io~projected/kube-api-access-qxbsz:{mountpoint:/var/lib/kubelet/pods/0aeb97e6-8dcd-4896-8a32-424681fcfc47/volumes/kubernetes.io~projected/kube-api-access-qxbsz major:0 minor:95 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a1760d4-777c-4853-bbbf-0a9afae8f45f/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2a1760d4-777c-4853-bbbf-0a9afae8f45f/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf/volumes/kubernetes.io~projected/kube-api-access-bb98m:{mountpoint:/var/lib/kubelet/pods/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf/volumes/kubernetes.io~projected/kube-api-access-bb98m major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf/volumes/kubernetes.io~secret/metrics-tls major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d584253-d14a-48e9-bfe6-563acb954e1f/volumes/kubernetes.io~projected/kube-api-access-d7bvw:{mountpoint:/var/lib/kubelet/pods/3d584253-d14a-48e9-bfe6-563acb954e1f/volumes/kubernetes.io~projected/kube-api-access-d7bvw major:0 minor:70 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/454cfe49-4029-4326-b0ce-f3fe9194d991/volumes/kubernetes.io~projected/kube-api-access-2hqnn:{mountpoint:/var/lib/kubelet/pods/454cfe49-4029-4326-b0ce-f3fe9194d991/volumes/kubernetes.io~projected/kube-api-access-2hqnn major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/454cfe49-4029-4326-b0ce-f3fe9194d991/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/454cfe49-4029-4326-b0ce-f3fe9194d991/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d53981b-d854-4b47-b1f1-f46b0c810830/volumes/kubernetes.io~projected/kube-api-access-4tz66:{mountpoint:/var/lib/kubelet/pods/4d53981b-d854-4b47-b1f1-f46b0c810830/volumes/kubernetes.io~projected/kube-api-access-4tz66 major:0 minor:71 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d53981b-d854-4b47-b1f1-f46b0c810830/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4d53981b-d854-4b47-b1f1-f46b0c810830/volumes/kubernetes.io~secret/metrics-certs major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b07fd43-f514-4533-9523-0f8f97930639/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7b07fd43-f514-4533-9523-0f8f97930639/volumes/kubernetes.io~secret/agent-certs major:0 minor:80 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:81 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~empty-dir/tmp major:0 minor:82 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~projected/kube-api-access-nh6c4:{mountpoint:/var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~projected/kube-api-access-nh6c4 major:0 minor:96 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/858636bd-15aa-47e2-95ce-daaf7cffb8d0/volumes/kubernetes.io~projected/kube-api-access-l27tt:{mountpoint:/var/lib/kubelet/pods/858636bd-15aa-47e2-95ce-daaf7cffb8d0/volumes/kubernetes.io~projected/kube-api-access-l27tt major:0 minor:97 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e/volumes/kubernetes.io~projected/kube-api-access-cl8zh:{mountpoint:/var/lib/kubelet/pods/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e/volumes/kubernetes.io~projected/kube-api-access-cl8zh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5f64145-a6c9-4d5c-8951-348943d7806c/volumes/kubernetes.io~projected/kube-api-access-bm7dt:{mountpoint:/var/lib/kubelet/pods/a5f64145-a6c9-4d5c-8951-348943d7806c/volumes/kubernetes.io~projected/kube-api-access-bm7dt major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c36f1c1a-8072-409c-ba64-929fe596778b/volumes/kubernetes.io~projected/kube-api-access-hxww8:{mountpoint:/var/lib/kubelet/pods/c36f1c1a-8072-409c-ba64-929fe596778b/volumes/kubernetes.io~projected/kube-api-access-hxww8 major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eaaec7db-74f2-49ac-a56b-198f78680fce/volumes/kubernetes.io~projected/kube-api-access-zsshs:{mountpoint:/var/lib/kubelet/pods/eaaec7db-74f2-49ac-a56b-198f78680fce/volumes/kubernetes.io~projected/kube-api-access-zsshs major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/253919c97492c3e0c537af073a474d67d2069827a6837c961052b41de6ad1896/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/dbfa4cc5b20ba8c2f1f7d45f985ed873b870144d9a6e3411a9dcdea3c70a7f75/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7168b56c3c2d0ced2fb4b81169793055bbcfee20d99ea80fe309e15371ce6be4/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-110:{mountpoint:/var/lib/containers/storage/overlay/4fdd8ca68c2639082b09f27c953b63211e503bc605dd7e8a90568b1e4a93dba1/merged major:0 minor:110 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/348dca51eb7c89f6b41b48f1f232c7ceda3850f0b499dd0510110c64f8ba3977/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3852c046f97f482c16255d4488f8286f25914018e7ce701f278fa3cb6ee129b0/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e28859ef7daaf2fa99fcf7be12b43e844bce1faf3e90d7738cf71270d1f206c9/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a479d7dc36a7704b1114c660eb7483b8040d3e975699cb1790f8e657f33a80dc/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/9924995f6175ec43360e33f460bb415f8517b81dbe6b253b706ef05404bf53c6/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0d741a0762238814ff79147213a28866a22357fcf0dea4ecb808ba273f6b9776/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/de0970f9aad8a43bd092407b0b8be8cd0b73f854a39068ad1518fd92dd189276/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/1caa362a6a5cb4ee3f9aa1622c1e3975dd9e4330c893cded1e1689a1d3ffdeb4/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-158:{mountpoint:/var/lib/containers/storage/overlay/bab7bcf717c8b76115d1aca38696a67e11463eb623193c456b88a5c9abfc833a/merged major:0 minor:158 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/04c1c424b57aea283b19f4bbffea0139cc0435da99cfc85d0035e12ab703b529/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/cfcfc1014b5717c430ca0c54ae5e031019704144ef7b971b37125578a46438d6/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-173:{mountpoint:/var/lib/containers/storage/overlay/c788b567ffff0f20fc904870919df90ee3cbe6be866831ea2fc8765796917dfa/merged major:0 minor:173 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/3735c141a46b07824ca38c8f6404f47790b2fe7ecac7bb92576f0c07ae44d1a8/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-183:{mountpoint:/var/lib/containers/storage/overlay/2b0deb53d78245967feb0c06d0901152875821170c0e2894087b483f512c1e20/merged major:0 minor:183 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/d7642fc1df46c99539bf20f17f3645d196623f8b2b83c24afcc0670aa0a0093c/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-193:{mountpoint:/var/lib/containers/storage/overlay/d57039164a24fe2ee1a03017ebd23abe4dc0527a43fea3a927ca369de084d63a/merged major:0 minor:193 fsType:overlay blockSize:0} overlay_0-198:{mountpoint:/var/lib/containers/storage/overlay/793b6d6fa34393549e4483867050e330eb5eee5f62e5b1d96396381d68ae329b/merged major:0 minor:198 fsType:overlay blockSize:0} overlay_0-200:{mountpoint:/var/lib/containers/storage/overlay/610e15365bc9af93234b17e22bed235c46eca61d6db787761e49f1a3bff0c158/merged major:0 minor:200 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/0587ec9befcd252057eb837f4cde7323e05145265233c142445ae6144df5a5ad/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/3b51067b1c861483b69105914d318170106934077281b07d19948632a8f4d70c/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/21bc72a6a55e12f144c72654e930d70ec283149fee2836f8b075894a8a6cdb30/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/2efeea3caa578c19ed8966a216c12bad23d2b581e3a3f93df1b4d534ef4f4c64/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/c88884fcded8b19e0433e983a801a242eeba8ae7b44aafda40faf5a5d623b7c1/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/6cf703c84920db04e66383f50d7b1b6ebc79e14d5945b81059570c5b16f97ecf/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d386bd8f6d90ba8f4b6128bf8582a018891c5d583e18409d4a92bb6c3e0889f0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f7fc853dff5f9b11c3372c2637405e8533bd91b39b27c719acd325a62348d692/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/abae1b52e3832d2093510f0bcdc23536816ce358d344793793e63963d371ac85/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/dee384792dc19413b412c610ed3e2f95fff0c678a9116aa2ad18b609d343505a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/d1abf3a24bf0c0bca82547a556c4913502ea08e702b62d0d40478c5b785c1bce/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/e259d6a712456497bf49e4ca6362ad355f1209031fd0bc5577cacc2ce4c8c7f1/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/a2a3737216715fd914f66f5a87fae6921619c8cccd87af787f7ea85b00b91753/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b1e8e7216bd5589eb01a8eacf8af4b2bc373ebffc35e56d2c5f4fe0a795194f5/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/c02286d254e6d14f7e4c6f7ea5f5ff1e467d0f249285d92836a09245ef094f4b/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/3e350f57f2d40c5c7da5820a62b0e233e1f50db955b19933707b7168c4a51cf8/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/515ec0476cb59fa08fa89147b2c7f04172ec02953f62f74feba8e9591fd696af/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/a6685d60fa53d679a2057adccae278512b0562589613526ab4f5b695c4945b6c/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/afcd55ff42a807b5a3191124bf9ddcf3d4662f1a093288ccc273c3b64e08586b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e73a4679ea3f5aaacf7de907f161e3e7ef22176ea67e35c2d66dbf79b16e659c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-55:{mountpoint:/var/lib/containers/storage/overlay/c9291f49374ca15bc7d786909059c8c146d39d337a7f9ff71f876d66d193c7eb/merged major:0 minor:55 fsType:overlay blockSize:0} overlay_0-58:{mountpoint:/var/lib/containers/storage/overlay/02f675d7d04e0fb5b94c87078f2cf52d036afcf18f9080a2ac50e3bbb92374b8/merged major:0 minor:58 fsType:overlay blockSize:0} overlay_0-61:{mountpoint:/var/lib/containers/storage/overlay/15db188e0ed3c8fdc405cd559e78de0a1c84fcd398e63d16b573bc1ada60d479/merged major:0 minor:61 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4cb262dea16d81fb7975dc815e704424b2e485c1ae47c6783d6386e398761a5b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/ef6b68fc0f4cb9c97e167e425c2f2579642fe248530bbb79f23ae154cd4241be/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/789e25fc16d67611a53463d78b64c1bee1420a4c2a58361d839355e32f99edf7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/476402a5f14b48998598c5ebe04293e4ea9d034990c3f71f3b62c3f0a63f7d83/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/757edd64278a230b6f33fae2790d2095339fb3b43ef1558ad10396c333e526ab/merged major:0 minor:93 fsType:overlay blockSize:0}]
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.500414    4888 manager.go:217] Machine: {Timestamp:2026-10-31 00:29:51.499955426 +0000 UTC m=+0.054842855 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3215119 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b89156c6c6480c8ff508e9a3ce798 SystemUUID:ec2b8915-6c6c-6480-c8ff-508e9a3ce798 BootID:00c39938-a3ea-4d24-b3bc-d4123b5391ac Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/454cfe49-4029-4326-b0ce-f3fe9194d991/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:213 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fee5b4908ed32bb9d799e56f477e2ae124cfcd74d46963364e061521c66025d9/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-58 DeviceMajor:0 DeviceMinor:58 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a1760d4-777c-4853-bbbf-0a9afae8f45f/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:247 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87caaa89b01db633825a45da0a845742cec1b15a776d8180df82f7dc6e9fc728/userdata/shm DeviceMajor:0 DeviceMinor:108 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:147 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-193 DeviceMajor:0 DeviceMinor:193 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/055635e6-bb27-4685-8f8e-f26d873567a8/volumes/kubernetes.io~projected/kube-api-access-mlj5w DeviceMajor:0 DeviceMinor:219 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a5f64145-a6c9-4d5c-8951-348943d7806c/volumes/kubernetes.io~projected/kube-api-access-bm7dt DeviceMajor:0 DeviceMinor:246 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e/volumes/kubernetes.io~projected/kube-api-access-cl8zh DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b6e21ce95b1d765a559ae1d4cccdf70c0f5e8451b3bac547fab779819a14737/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volumes/kubernetes.io~projected/kube-api-access-bm2hb DeviceMajor:0 DeviceMinor:68 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c36f1c1a-8072-409c-ba64-929fe596778b/volumes/kubernetes.io~projected/kube-api-access-hxww8 DeviceMajor:0 DeviceMinor:69 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1223b71cf0a603c934d3f9c07558f7c8a570f784f970a554124ed3bc3020caae/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0aeb97e6-8dcd-4896-8a32-424681fcfc47/volumes/kubernetes.io~projected/kube-api-access-qxbsz DeviceMajor:0 DeviceMinor:95 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf/volumes/kubernetes.io~projected/kube-api-access-bb98m DeviceMajor:0 DeviceMinor:220 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4b3f141556a418cea1aab93857408463bb46f735d0c768878b7f321ff25a36d5/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af99bd996b2d98a9d885edc409f4aa9dab998e5c29f615601fafce63b12dcb72/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:81 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-200 DeviceMajor:0 DeviceMinor:200 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32adff65999d9cdc6af88443bc0074345790dc83bd70c2344f52fe4279ceff50/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-158 DeviceMajor:0 DeviceMinor:158 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6843b645ac49edfc71489a9ed5110c48402c455d35ebb81e86185affa9e03652/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d392df2bf42859cf3facac13cbe5f1b870941235d40ddb1482e793fb18b275d9/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01df07af-c80e-4ae5-950d-87cc511cbda7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/454cfe49-4029-4326-b0ce-f3fe9194d991/volumes/kubernetes.io~projected/kube-api-access-2hqnn DeviceMajor:0 DeviceMinor:221 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06a421cfd426cf060983d93846152acb56e9cb14ba8f96e860b5cbb13ce391ec/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b07fd43-f514-4533-9523-0f8f97930639/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:80 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-55 DeviceMajor:0 DeviceMinor:55 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d53981b-d854-4b47-b1f1-f46b0c810830/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-61 DeviceMajor:0 DeviceMinor:61 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:82 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8d2e85eca66a1429f17d4cbc5e4f7613bf3e15d0017439c4a2a7991ef0bdd08/userdata/shm DeviceMajor:0 DeviceMinor:102 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-173 DeviceMajor:0 DeviceMinor:173 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-198 DeviceMajor:0 DeviceMinor:198 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/055635e6-bb27-4685-8f8e-f26d873567a8/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/11eb5d7869d8477964c2140614c7d3f0c2f9c2ee4f8f19b690c7f218c68e2fba/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/eaaec7db-74f2-49ac-a56b-198f78680fce/volumes/kubernetes.io~projected/kube-api-access-zsshs DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f898bf492e7ce0e4cd1003a6c9536d8f2f903bc7d8d29cc122aa5b41dcefe8e/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:218 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6f65e92c3e360bce2705017fb50ebc484044bf53586582f1bb3596986e330c7/userdata/shm DeviceMajor:0 DeviceMinor:142 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-110 DeviceMajor:0 DeviceMinor:110 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a2d8ef9d79978edb55e4c9a03cc42df2ec6e59d5ed7176f67d0cc2199c7fd2e/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d584253-d14a-48e9-bfe6-563acb954e1f/volumes/kubernetes.io~projected/kube-api-access-d7bvw DeviceMajor:0 DeviceMinor:70 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d53981b-d854-4b47-b1f1-f46b0c810830/volumes/kubernetes.io~projected/kube-api-access-4tz66 DeviceMajor:0 DeviceMinor:71 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7c477db4-252e-4236-9ddd-79307115124f/volumes/kubernetes.io~projected/kube-api-access-nh6c4 DeviceMajor:0 DeviceMinor:96 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/858636bd-15aa-47e2-95ce-daaf7cffb8d0/volumes/kubernetes.io~projected/kube-api-access-l27tt DeviceMajor:0 DeviceMinor:97 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:150 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02ae9325-803d-4acc-bafe-4349f1548558/volumes/kubernetes.io~projected/kube-api-access-4pgxr DeviceMajor:0 DeviceMinor:151 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-183 DeviceMajor:0 DeviceMinor:183 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:11eb5d7869d8477 MacAddress:46:17:a5:4c:a6:b9 Speed:10000 Mtu:8901} {Name:2a2d8ef9d79978e MacAddress:82:92:02:40:11:5e Speed:10000 Mtu:8901} {Name:4b3f141556a418c MacAddress:16:55:45:8f:d2:b6 Speed:10000 Mtu:8901} {Name:7b6e21ce95b1d76 MacAddress:4e:f8:01:f0:72:de Speed:10000 Mtu:8901} {Name:af99bd996b2d98a MacAddress:26:a5:c9:25:02:a4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cf:f3:51:0d Speed:0 Mtu:9001} {Name:br-int MacAddress:f2:8f:1b:61:e3:57 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:cf:f3:51:0d Speed:0 Mtu:9001} {Name:fee5b4908ed32bb MacAddress:9a:2b:26:6d:8d:9f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:f6:9a:31:7d:a6:dd Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:b4:51:04:88:42 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.500670    4888 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.500709    4888 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.501508    4888 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.501553    4888 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-97.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.501978    4888 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.501991    4888 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502004    4888 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502032    4888 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:29:51.502413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502069    4888 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:29:51.502926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502542    4888 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:29:51.502926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502654    4888 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:29:51.502926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502688    4888 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:29:51.502926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502701    4888 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:29:51.502926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502712    4888 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:29:51.502926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502720    4888 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:29:51.502926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.502728    4888 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:29:51.503287 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.503273    4888 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:29:51.503287 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.503286    4888 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:29:51.503762 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.503742    4888 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:29:51.505648 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.505629    4888 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:29:51.505648 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.505642    4888 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:29:51.505787 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.505729    4888 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 00:29:51.505918 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.505907    4888 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:29:51.505948 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.505935    4888 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:29:51.506029 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506019    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:29:51.506067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506034    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:29:51.506067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506040    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:29:51.506067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506045    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:29:51.506067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506050    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:29:51.506067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506056    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:29:51.506067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506061    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:29:51.506067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506066    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:29:51.506295 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506073    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:29:51.506295 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506078    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:29:51.506295 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506091    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:29:51.506295 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506100    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:29:51.506295 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506125    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:29:51.506295 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.506131    4888 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:29:51.509254 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.509242    4888 server.go:1263] "Started kubelet"
Oct 31 00:29:51.509336 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.509305    4888 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:29:51.509397 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.509332    4888 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:29:51.509441 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.509420    4888 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:29:51.509837 ip-10-0-131-97 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:29:51.509990 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.509927    4888 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:29:51.513451 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.513101    4888 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:29:51.514190 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.514168    4888 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:29:51.515166 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.514785    4888 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:29:51.515166 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.514949    4888 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:29:51.515166 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.514963    4888 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:29:51.515166 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.514956    4888 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 00:24:13 +0000 UTC" deadline="2028-08-11 19:24:33.442622738 +0000 UTC"
Oct 31 00:29:51.515166 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.514995    4888 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15618h54m41.927631245s"
Oct 31 00:29:51.515166 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.515116    4888 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found"
Oct 31 00:29:51.516377 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.516253    4888 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:29:51.516688 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.516671    4888 factory.go:55] Registering systemd factory
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.516691    4888 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517034    4888 factory.go:153] Registering CRI-O factory
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517048    4888 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517092    4888 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517110    4888 factory.go:103] Registering Raw factory
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517124    4888 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517480    4888 manager.go:319] Starting recovery of all containers
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517916    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01df07af-c80e-4ae5-950d-87cc511cbda7" volumeName="kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517962    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb97e6-8dcd-4896-8a32-424681fcfc47" volumeName="kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-daemon-config" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517978    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2add7e6b-c8b7-4983-952d-9ff9c90b5bdf" volumeName="kubernetes.io/projected/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-kube-api-access-bb98m" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.517992    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01df07af-c80e-4ae5-950d-87cc511cbda7" volumeName="kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-env-overrides" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518005    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2add7e6b-c8b7-4983-952d-9ff9c90b5bdf" volumeName="kubernetes.io/empty-dir/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-tmp-dir" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518018    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b07fd43-f514-4533-9523-0f8f97930639" volumeName="kubernetes.io/configmap/7b07fd43-f514-4533-9523-0f8f97930639-konnectivity-ca" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518031    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c477db4-252e-4236-9ddd-79307115124f" volumeName="kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-etc-tuned" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518044    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d66308f-554a-4ea4-91ed-ffe42bbb0f3e" volumeName="kubernetes.io/configmap/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-serviceca" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518060    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01df07af-c80e-4ae5-950d-87cc511cbda7" volumeName="kubernetes.io/projected/01df07af-c80e-4ae5-950d-87cc511cbda7-kube-api-access-bm2hb" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518074    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ae9325-803d-4acc-bafe-4349f1548558" volumeName="kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518088    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ae9325-803d-4acc-bafe-4349f1548558" volumeName="kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-tls" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518102    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="055635e6-bb27-4685-8f8e-f26d873567a8" volumeName="kubernetes.io/secret/055635e6-bb27-4685-8f8e-f26d873567a8-cert" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518114    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2add7e6b-c8b7-4983-952d-9ff9c90b5bdf" volumeName="kubernetes.io/secret/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-metrics-tls" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518126    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858636bd-15aa-47e2-95ce-daaf7cffb8d0" volumeName="kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518141    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454cfe49-4029-4326-b0ce-f3fe9194d991" volumeName="kubernetes.io/projected/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-api-access-2hqnn" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518153    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454cfe49-4029-4326-b0ce-f3fe9194d991" volumeName="kubernetes.io/secret/454cfe49-4029-4326-b0ce-f3fe9194d991-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518165    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d53981b-d854-4b47-b1f1-f46b0c810830" volumeName="kubernetes.io/projected/4d53981b-d854-4b47-b1f1-f46b0c810830-kube-api-access-4tz66" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518179    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01df07af-c80e-4ae5-950d-87cc511cbda7" volumeName="kubernetes.io/secret/01df07af-c80e-4ae5-950d-87cc511cbda7-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518191    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ae9325-803d-4acc-bafe-4349f1548558" volumeName="kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518203    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ae9325-803d-4acc-bafe-4349f1548558" volumeName="kubernetes.io/empty-dir/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-textfile" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518216    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a1760d4-777c-4853-bbbf-0a9afae8f45f" volumeName="kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518227    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2add7e6b-c8b7-4983-952d-9ff9c90b5bdf" volumeName="kubernetes.io/configmap/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-config-volume" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518246    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="055635e6-bb27-4685-8f8e-f26d873567a8" volumeName="kubernetes.io/projected/055635e6-bb27-4685-8f8e-f26d873567a8-kube-api-access-mlj5w" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518259    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d584253-d14a-48e9-bfe6-563acb954e1f" volumeName="kubernetes.io/projected/3d584253-d14a-48e9-bfe6-563acb954e1f-kube-api-access-d7bvw" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518271    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d53981b-d854-4b47-b1f1-f46b0c810830" volumeName="kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518282    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454cfe49-4029-4326-b0ce-f3fe9194d991" volumeName="kubernetes.io/configmap/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518292    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="454cfe49-4029-4326-b0ce-f3fe9194d991" volumeName="kubernetes.io/empty-dir/454cfe49-4029-4326-b0ce-f3fe9194d991-data-volume" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518320    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b07fd43-f514-4533-9523-0f8f97930639" volumeName="kubernetes.io/secret/7b07fd43-f514-4533-9523-0f8f97930639-agent-certs" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518336    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c477db4-252e-4236-9ddd-79307115124f" volumeName="kubernetes.io/projected/7c477db4-252e-4236-9ddd-79307115124f-kube-api-access-nh6c4" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518365    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858636bd-15aa-47e2-95ce-daaf7cffb8d0" volumeName="kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518390    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ae9325-803d-4acc-bafe-4349f1548558" volumeName="kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-metrics-client-ca" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518418    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c477db4-252e-4236-9ddd-79307115124f" volumeName="kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-tmp" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518429    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858636bd-15aa-47e2-95ce-daaf7cffb8d0" volumeName="kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518439    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="858636bd-15aa-47e2-95ce-daaf7cffb8d0" volumeName="kubernetes.io/projected/858636bd-15aa-47e2-95ce-daaf7cffb8d0-kube-api-access-l27tt" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518450    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c36f1c1a-8072-409c-ba64-929fe596778b" volumeName="kubernetes.io/empty-dir/c36f1c1a-8072-409c-ba64-929fe596778b-tmp-dir" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518461    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c36f1c1a-8072-409c-ba64-929fe596778b" volumeName="kubernetes.io/projected/c36f1c1a-8072-409c-ba64-929fe596778b-kube-api-access-hxww8" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518472    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaaec7db-74f2-49ac-a56b-198f78680fce" volumeName="kubernetes.io/configmap/eaaec7db-74f2-49ac-a56b-198f78680fce-iptables-alerter-script" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518482    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ae9325-803d-4acc-bafe-4349f1548558" volumeName="kubernetes.io/projected/02ae9325-803d-4acc-bafe-4349f1548558-kube-api-access-4pgxr" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518495    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb97e6-8dcd-4896-8a32-424681fcfc47" volumeName="kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518506    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0aeb97e6-8dcd-4896-8a32-424681fcfc47" volumeName="kubernetes.io/projected/0aeb97e6-8dcd-4896-8a32-424681fcfc47-kube-api-access-qxbsz" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518518    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d66308f-554a-4ea4-91ed-ffe42bbb0f3e" volumeName="kubernetes.io/projected/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-kube-api-access-cl8zh" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518530    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eaaec7db-74f2-49ac-a56b-198f78680fce" volumeName="kubernetes.io/projected/eaaec7db-74f2-49ac-a56b-198f78680fce-kube-api-access-zsshs" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518542    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01df07af-c80e-4ae5-950d-87cc511cbda7" volumeName="kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-config" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518554    4888 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5f64145-a6c9-4d5c-8951-348943d7806c" volumeName="kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt" seLinuxMountContext=""
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518565    4888 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:29:51.518840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.518573    4888 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:29:51.541140 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.541117    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:29:51.543515 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.543497    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:29:51.545657 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.545451    4888 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:29:51.547394 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.547372    4888 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:29:51.547463 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.547408    4888 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:29:51.547463 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.547430    4888 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:29:51.547564 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.547470    4888 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 00:29:51.563837 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.563813    4888 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="c82599d354f3cf9c16ba99048f723adc4c23211b17f0e2fabc7a4f01feadfec4" exitCode=0
Oct 31 00:29:51.563837 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.563837    4888 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="857cad3e8ce048b345cc95ba2617a455be93369d8b46182dbafe08527a1478d3" exitCode=0
Oct 31 00:29:51.563962 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.563844    4888 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="c104edc53bd835027090be0e36b121eefc1210ffa6c966cea962eaca16b968b6" exitCode=0
Oct 31 00:29:51.563962 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.563849    4888 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="4649aea2684dcd2226eb65d631da18548b2c7b246a1874f2bc6795a07a135e3c" exitCode=0
Oct 31 00:29:51.563962 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.563854    4888 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="6f1d5fdc3d44f6548e93e74a0e6a76dd9940b7da650235db94b346bd8998bd07" exitCode=0
Oct 31 00:29:51.563962 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.563861    4888 generic.go:356] "Generic (PLEG): container finished" podID="858636bd-15aa-47e2-95ce-daaf7cffb8d0" containerID="049ded342751a7a0693a424c1b7c38c7c0ef81e18bd0dd528b7eb8ad934d791e" exitCode=0
Oct 31 00:29:51.565493 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.565466    4888 generic.go:356] "Generic (PLEG): container finished" podID="e8796dcae988b291465b4878c3ad6567" containerID="4bbc3bf450777b45be3094e5fc80092834b483e254753859328a191a79454bdc" exitCode=0
Oct 31 00:29:51.568942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.568920    4888 generic.go:356] "Generic (PLEG): container finished" podID="02ae9325-803d-4acc-bafe-4349f1548558" containerID="15082ddd300228ad2b9b2c9c6720cb7a69eb89b3849ec6d7d31f73b1bf6fb097" exitCode=0
Oct 31 00:29:51.582522 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.582506    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:29:51.583153 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.583132    4888 generic.go:356] "Generic (PLEG): container finished" podID="3d584253-d14a-48e9-bfe6-563acb954e1f" containerID="0e74c1c16c2a903f617696d0950c61d8572f4ed37fde6d1a9cf08c0059d66fbb" exitCode=1
Oct 31 00:29:51.593758 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.593741    4888 manager.go:324] Recovery completed
Oct 31 00:29:51.609449 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.609431    4888 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:51.610422 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.610408    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:51.610481 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.610435    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:51.610481 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.610445    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:51.612864 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.612849    4888 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:29:51.612864 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.612862    4888 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:29:51.612964 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.612868    4888 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:51.612964 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.612880    4888 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:29:51.613041 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613027    4888 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 00:29:51.613070 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613043    4888 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 00:29:51.613070 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613059    4888 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 00:29:51.613070 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613064    4888 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 00:29:51.613070 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613069    4888 policy_none.go:50] "Start"
Oct 31 00:29:51.613207 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613077    4888 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:29:51.613207 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613088    4888 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:29:51.613291 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613221    4888 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 00:29:51.613291 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613228    4888 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 00:29:51.613291 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613233    4888 policy_none.go:44] "Start"
Oct 31 00:29:51.613730 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613713    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:51.613786 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613748    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:51.613786 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.613764    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:51.615314 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.615294    4888 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found"
Oct 31 00:29:51.617128 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.617104    4888 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:29:51.620557 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620542    4888 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:29:51.620649 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.620566    4888 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:29:51.620649 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620576    4888 server.go:87] "Starting device plugin registration server"
Oct 31 00:29:51.620804 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620721    4888 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:29:51.620804 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620731    4888 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:29:51.620897 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620807    4888 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:29:51.620941 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620882    4888 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:29:51.620941 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620913    4888 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:29:51.620941 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620918    4888 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:29:51.621074 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.620978    4888 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:29:51.620905168Z","UUID":"06041686-7945-4377-9da6-8b60c94523d3","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:29:51.622444 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.622337    4888 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:29:51.622444 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.622394    4888 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-97.us-west-2.compute.internal\" not found"
Oct 31 00:29:51.623491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.623476    4888 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:29:51.623580 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.623499    4888 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:29:51.648302 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.647856    4888 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"]
Oct 31 00:29:51.648409 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.648398    4888 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:51.649085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.649070    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:51.649159 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.649099    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:51.649159 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.649113    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:51.649390 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.649377    4888 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:51.649738 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.649721    4888 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:51.650525 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.650505    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:51.650598 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.650536    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:51.650598 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.650556    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:51.650740 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.650723    4888 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.650897 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.650884    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:51.650944 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.650913    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:51.651001 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.650944    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:51.651402 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.651340    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerStarted","Data":"82503f93adfec75ac05daecb881ea73b7d2d9555a3f5843dca9ceb8f8a07bb63"}
Oct 31 00:29:51.651467 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.651420    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerDied","Data":"4bbc3bf450777b45be3094e5fc80092834b483e254753859328a191a79454bdc"}
Oct 31 00:29:51.651467 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.651451    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"e8796dcae988b291465b4878c3ad6567","Type":"ContainerStarted","Data":"99aa908a9bcbf37f86738e306d07cef31a9222ae9ba882cbcbc691d225f57199"}
Oct 31 00:29:51.651549 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.651470    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"0b932d77d38edb2300c0d4d58dae2752","Type":"ContainerStarted","Data":"7f4c98fe5dd5b033d354ae3a78949a43f6290ab2673dc7865dbf458810fc7f9f"}
Oct 31 00:29:51.651549 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.651494    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal" event={"ID":"0b932d77d38edb2300c0d4d58dae2752","Type":"ContainerStarted","Data":"389b739d734b365c56ba3e8617d4d3405e160047837552df9960e33eab6e7a7d"}
Oct 31 00:29:51.651678 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.651660    4888 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:51.652733 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.652690    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:51.652733 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.652719    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:51.652733 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.652729    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:51.652861 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.652840    4888 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-97.us-west-2.compute.internal\" not found" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.702176 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.702153    4888 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:51.702266 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.702187    4888 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:51.702302 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.702295    4888 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 00:29:51.703921 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.703907    4888 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:51.715991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.715973    4888 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.723220 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.723205    4888 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:29:51.723951 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.723939    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:29:51.724021 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.723965    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:29:51.724021 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.723976    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:29:51.724021 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.724004    4888 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.795904 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.795887    4888 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:29:51.795983 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.795902    4888 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:29:51.795983 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.795969    4888 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.796065 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.795984    4888 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.803103 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.803087    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.803144 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.803110    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.803144 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.803128    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b932d77d38edb2300c0d4d58dae2752-config\") pod \"kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"0b932d77d38edb2300c0d4d58dae2752\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.845884 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.845868    4888 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.845945 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.845922    4888 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.846404 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.846390    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeNotReady"
Oct 31 00:29:51.846450 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.846416    4888 setters.go:546] "Node became not ready" node="ip-10-0-131-97.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T00:29:51Z","lastTransitionTime":"2026-10-31T00:29:51Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 00:29:51.877926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.877909    4888 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:29:51.877998 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:51.877943    4888 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.903972 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.903924    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.903972 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.903950    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.903972 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.903967    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b932d77d38edb2300c0d4d58dae2752-config\") pod \"kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"0b932d77d38edb2300c0d4d58dae2752\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.904081 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.904012    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0b932d77d38edb2300c0d4d58dae2752-config\") pod \"kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"0b932d77d38edb2300c0d4d58dae2752\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.904081 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.904037    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.904081 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.904038    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8796dcae988b291465b4878c3ad6567-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" (UID: \"e8796dcae988b291465b4878c3ad6567\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:51.905872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:51.905858    4888 nodeinfomanager.go:471] existing CSINode "ip-10-0-131-97.us-west-2.compute.internal" is owned by different node (oldNodeID="71385b99-ab13-4269-ba93-631393927193", newNodeID=""), cleaning up...
Oct 31 00:29:52.503062 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.503043    4888 apiserver.go:52] "Watching apiserver"
Oct 31 00:29:52.593178 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.593160    4888 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:52.602194 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.602175    4888 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:29:52.602636 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.602617    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-knvdm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-cntts","openshift-monitoring/node-exporter-9cnsr","openshift-ovn-kubernetes/ovnkube-node-gxlg7","openshift-cluster-node-tuning-operator/tuned-2rxnb","openshift-dns/node-resolver-wjjv6","openshift-multus/multus-jh76v","kube-system/kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-mr44j","kube-system/global-pull-secret-syncer-hrxqv","openshift-image-registry/node-ca-j9jsc","openshift-multus/network-metrics-daemon-2fk4z","openshift-dns/dns-default-kdkdw","openshift-insights/insights-runtime-extractor-nm5ww","openshift-multus/multus-additional-cni-plugins-vx9mh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn","kube-system/konnectivity-agent-gmcv9"]
Oct 31 00:29:52.617194 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.617168    4888 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:29:52.652164 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.652147    4888 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:29:52.652214 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:29:52.652183    4888 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-97.us-west-2.compute.internal"
Oct 31 00:29:52.655093 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655072    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.655366 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655329    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:29:52.655366 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655343    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.655631 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655614    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.655703 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655626    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.655703 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655651    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.655784 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655660    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.655784 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655713    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bv698\""
Oct 31 00:29:52.655858 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655843    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4l8km\""
Oct 31 00:29:52.655909 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655896    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.655955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655905    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s5c24\""
Oct 31 00:29:52.655955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655931    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:29:52.655955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655943    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.655955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655945    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5b488\""
Oct 31 00:29:52.655955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655936    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:29:52.656128 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.655943    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.656174 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656134    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vzctz\""
Oct 31 00:29:52.656174 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656138    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.656174 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656156    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.656174 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656163    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:29:52.656174 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656156    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:29:52.656343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656156    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:29:52.656343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656184    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:29:52.656343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656157    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:29:52.656343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656233    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.656343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656323    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6xpx\""
Oct 31 00:29:52.656513 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656430    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:29:52.656513 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656479    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:29:52.656611 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.656600    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l8m77\""
Oct 31 00:29:52.661641 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.661626    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.708860 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.708806    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-tmp\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.708860 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.708842    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.708966 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.708888    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.708966 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.708890    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-tmp\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.708966 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.708913    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:52.708966 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.708959    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-systemd-units\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709121 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709028    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709121 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709076    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-modprobe-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.709200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709116    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-kubernetes\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.709200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709145    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-systemd\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.709200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709175    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-etc-tuned\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.709311 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709239    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.709311 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709250    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7c477db4-252e-4236-9ddd-79307115124f-etc-tuned\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.709311 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709274    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cni-binary-copy\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.709311 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709305    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tz66\" (UniqueName: \"kubernetes.io/projected/4d53981b-d854-4b47-b1f1-f46b0c810830-kube-api-access-4tz66\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:52.709464 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709450    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fd43-f514-4533-9523-0f8f97930639-konnectivity-ca\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:52.709513 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709475    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01df07af-c80e-4ae5-950d-87cc511cbda7-ovn-node-metrics-cert\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709513 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709496    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-script-lib\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709605 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709530    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.709605 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709558    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-serviceca\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:52.709605 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709585    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cni-binary-copy\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.709748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709674    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-host\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.709748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709711    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-bin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.709748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709728    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/01df07af-c80e-4ae5-950d-87cc511cbda7-ovn-node-metrics-cert\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709749    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-systemd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709758    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-script-lib\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709796    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-ovn\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.709872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709815    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-serviceca\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:52.709991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709865    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-var-lib-kubelet\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.709991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709891    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-system-cni-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.709991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709907    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.709991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709939    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cnibin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.709991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709959    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-kubelet\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.709991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709982    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-daemon-config\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.710149 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.709998    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-node-log\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.710149 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710103    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8zh\" (UniqueName: \"kubernetes.io/projected/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-kube-api-access-cl8zh\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:52.710149 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710137    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cnibin\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.710247 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710163    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-multus\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.710247 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710170    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-daemon-config\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.710247 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710188    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-multus-certs\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.710247 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710228    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-wtmp\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710247    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710262    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-tmp-dir\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710282    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710297    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bvw\" (UniqueName: \"kubernetes.io/projected/3d584253-d14a-48e9-bfe6-563acb954e1f-kube-api-access-d7bvw\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710316    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-socket-dir-parent\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710360    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlj5w\" (UniqueName: \"kubernetes.io/projected/055635e6-bb27-4685-8f8e-f26d873567a8-kube-api-access-mlj5w\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710372    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-tmp-dir\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.710410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710391    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-sys\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710484    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb98m\" (UniqueName: \"kubernetes.io/projected/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-kube-api-access-bb98m\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710515    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-host\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710533    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710550    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-kubelet\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710565    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm2hb\" (UniqueName: \"kubernetes.io/projected/01df07af-c80e-4ae5-950d-87cc511cbda7-kube-api-access-bm2hb\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710589    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-sys\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710683    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-system-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.710725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710715    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-root\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710761    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-kubelet-config\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710792    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454cfe49-4029-4326-b0ce-f3fe9194d991-crio-socket\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710815    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hqnn\" (UniqueName: \"kubernetes.io/projected/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-api-access-2hqnn\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710830    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710845    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710866    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-hostroot\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710867    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710890    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055635e6-bb27-4685-8f8e-f26d873567a8-cert\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:52.710979 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710913    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-config\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.710997    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-env-overrides\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711053    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c36f1c1a-8072-409c-ba64-929fe596778b-hosts-file\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711067    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711108    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-ovnkube-config\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711112    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36f1c1a-8072-409c-ba64-929fe596778b-tmp-dir\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711148    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-run\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711163    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c36f1c1a-8072-409c-ba64-929fe596778b-tmp-dir\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711176    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711196    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/01df07af-c80e-4ae5-950d-87cc511cbda7-env-overrides\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711203    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxbsz\" (UniqueName: \"kubernetes.io/projected/0aeb97e6-8dcd-4896-8a32-424681fcfc47-kube-api-access-qxbsz\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711230    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaec7db-74f2-49ac-a56b-198f78680fce-host-slash\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:52.711268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711253    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-log-socket\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711279    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxww8\" (UniqueName: \"kubernetes.io/projected/c36f1c1a-8072-409c-ba64-929fe596778b-kube-api-access-hxww8\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711308    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysconfig\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711329    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/858636bd-15aa-47e2-95ce-daaf7cffb8d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711392    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-metrics-client-ca\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711437    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711462    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711487    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-textfile\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711504    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-tls\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711519    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pgxr\" (UniqueName: \"kubernetes.io/projected/02ae9325-803d-4acc-bafe-4349f1548558-kube-api-access-4pgxr\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711536    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-config-volume\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711558    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-metrics-tls\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711580    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-slash\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711585    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-textfile\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711602    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-netns\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.711692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711651    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh6c4\" (UniqueName: \"kubernetes.io/projected/7c477db4-252e-4236-9ddd-79307115124f-kube-api-access-nh6c4\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711750    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l27tt\" (UniqueName: \"kubernetes.io/projected/858636bd-15aa-47e2-95ce-daaf7cffb8d0-kube-api-access-l27tt\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711778    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-netns\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711802    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b07fd43-f514-4533-9523-0f8f97930639-agent-certs\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711839    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-etc-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711864    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-bin\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711895    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-netd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711926    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454cfe49-4029-4326-b0ce-f3fe9194d991-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711955    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-device-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.711971    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-conf-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712011    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712035    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712085    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-conf\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712120    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-lib-modules\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.712133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712138    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-etc-kubernetes\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712170    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaaec7db-74f2-49ac-a56b-198f78680fce-iptables-alerter-script\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712187    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-var-lib-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712202    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454cfe49-4029-4326-b0ce-f3fe9194d991-data-volume\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712216    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712248    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-k8s-cni-cncf-io\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712268    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsshs\" (UniqueName: \"kubernetes.io/projected/eaaec7db-74f2-49ac-a56b-198f78680fce-kube-api-access-zsshs\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712277    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/454cfe49-4029-4326-b0ce-f3fe9194d991-data-volume\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712284    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712308    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.712529 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.712327    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-dbus\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.755140 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755115    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:29:52.755254 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755185    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.755404 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755387    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.755498 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755483    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:29:52.755749 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755723    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.755852 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755796    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.755852 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755843    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.755955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755859    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.755955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755907    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.756032 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755974    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:29:52.756032 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.755994    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:29:52.756115 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756033    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:29:52.756115 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756101    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 00:29:52.756234 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756220    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 00:29:52.756277 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756224    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:29:52.756318 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756284    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:29:52.756318 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756309    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9b4md\""
Oct 31 00:29:52.756417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756322    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 00:29:52.756417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756394    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 00:29:52.756508 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756444    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-klgwd\""
Oct 31 00:29:52.756508 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756453    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-6z87r\""
Oct 31 00:29:52.756508 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756494    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:29:52.756621 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756508    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:29:52.756621 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756510    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mqw2c\""
Oct 31 00:29:52.756621 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756562    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:29:52.756724 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756635    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-prxzz\""
Oct 31 00:29:52.756792 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756771    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:29:52.756837 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756807    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:29:52.756878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756840    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-bpzsc\""
Oct 31 00:29:52.756878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756850    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2sfqr\""
Oct 31 00:29:52.756966 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756946    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5vb9k\""
Oct 31 00:29:52.757011 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.756998    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wzhm5\""
Oct 31 00:29:52.759867 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.759851    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a1760d4-777c-4853-bbbf-0a9afae8f45f-original-pull-secret\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.759911 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.759878    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7b07fd43-f514-4533-9523-0f8f97930639-konnectivity-ca\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:52.761054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.761037    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.761197 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.761182    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055635e6-bb27-4685-8f8e-f26d873567a8-cert\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:52.762208 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762183    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-metrics-tls\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.762294 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762208    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/454cfe49-4029-4326-b0ce-f3fe9194d991-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.762294 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762249    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-tls\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.762294 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762265    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7b07fd43-f514-4533-9523-0f8f97930639-agent-certs\") pod \"konnectivity-agent-gmcv9\" (UID: \"7b07fd43-f514-4533-9523-0f8f97930639\") " pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:29:52.762294 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762186    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-config-volume\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.762486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762334    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.762486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762412    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-accelerators-collector-config\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.762486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762449    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02ae9325-803d-4acc-bafe-4349f1548558-metrics-client-ca\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.762486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762481    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d53981b-d854-4b47-b1f1-f46b0c810830-metrics-certs\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:52.762599 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.762542    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/eaaec7db-74f2-49ac-a56b-198f78680fce-iptables-alerter-script\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:52.766608 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.766592    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm7dt\" (UniqueName: \"kubernetes.io/projected/a5f64145-a6c9-4d5c-8951-348943d7806c-kube-api-access-bm7dt\") pod \"network-check-target-mr44j\" (UID: \"a5f64145-a6c9-4d5c-8951-348943d7806c\") " pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:29:52.766714 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.766700    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tz66\" (UniqueName: \"kubernetes.io/projected/4d53981b-d854-4b47-b1f1-f46b0c810830-kube-api-access-4tz66\") pod \"network-metrics-daemon-2fk4z\" (UID: \"4d53981b-d854-4b47-b1f1-f46b0c810830\") " pod="openshift-multus/network-metrics-daemon-2fk4z"
Oct 31 00:29:52.767448 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.767435    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb98m\" (UniqueName: \"kubernetes.io/projected/2add7e6b-c8b7-4983-952d-9ff9c90b5bdf-kube-api-access-bb98m\") pod \"dns-default-kdkdw\" (UID: \"2add7e6b-c8b7-4983-952d-9ff9c90b5bdf\") " pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:52.767486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.767446    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8zh\" (UniqueName: \"kubernetes.io/projected/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-kube-api-access-cl8zh\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:52.769500 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.769484    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsshs\" (UniqueName: \"kubernetes.io/projected/eaaec7db-74f2-49ac-a56b-198f78680fce-kube-api-access-zsshs\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:52.769571 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.769557    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlj5w\" (UniqueName: \"kubernetes.io/projected/055635e6-bb27-4685-8f8e-f26d873567a8-kube-api-access-mlj5w\") pod \"ingress-canary-knvdm\" (UID: \"055635e6-bb27-4685-8f8e-f26d873567a8\") " pod="openshift-ingress-canary/ingress-canary-knvdm"
Oct 31 00:29:52.770279 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.770264    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bvw\" (UniqueName: \"kubernetes.io/projected/3d584253-d14a-48e9-bfe6-563acb954e1f-kube-api-access-d7bvw\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.770437 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.770422    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxbsz\" (UniqueName: \"kubernetes.io/projected/0aeb97e6-8dcd-4896-8a32-424681fcfc47-kube-api-access-qxbsz\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.772385 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.772370    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm2hb\" (UniqueName: \"kubernetes.io/projected/01df07af-c80e-4ae5-950d-87cc511cbda7-kube-api-access-bm2hb\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.772863 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.772848    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pgxr\" (UniqueName: \"kubernetes.io/projected/02ae9325-803d-4acc-bafe-4349f1548558-kube-api-access-4pgxr\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.773047 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.773028    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l27tt\" (UniqueName: \"kubernetes.io/projected/858636bd-15aa-47e2-95ce-daaf7cffb8d0-kube-api-access-l27tt\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.773155 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.773143    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh6c4\" (UniqueName: \"kubernetes.io/projected/7c477db4-252e-4236-9ddd-79307115124f-kube-api-access-nh6c4\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.774694 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.774680    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hqnn\" (UniqueName: \"kubernetes.io/projected/454cfe49-4029-4326-b0ce-f3fe9194d991-kube-api-access-2hqnn\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.774818 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.774800    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxww8\" (UniqueName: \"kubernetes.io/projected/c36f1c1a-8072-409c-ba64-929fe596778b-kube-api-access-hxww8\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:52.813578 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813559    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-netns\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.813633 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813583    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-netns\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.813633 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813601    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-etc-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.813633 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813614    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-bin\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813629    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-netd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813661    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-device-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813670    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-netns\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813685    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-conf-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813670    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-netns\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813694    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-bin\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813672    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-etc-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813718    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-conf-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.813746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813718    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-cni-netd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813709    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-conf\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813785    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-lib-modules\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813810    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-etc-kubernetes\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813836    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-var-lib-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813843    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-etc-kubernetes\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813813    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-conf\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813860    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813727    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-device-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813873    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-lib-modules\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813892    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-var-lib-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813916    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-socket-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813918    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-k8s-cni-cncf-io\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813944    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-k8s-cni-cncf-io\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.813959    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814002    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-dbus\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814043    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814095    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.814150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814143    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814148    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814170    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-systemd-units\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814145    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-dbus\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814192    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814200    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814212    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-systemd-units\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814239    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-modprobe-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814253    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-kubernetes\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814285    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-openvswitch\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814322    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-kubernetes\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814327    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-systemd\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814365    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814384    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-modprobe-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814392    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-host\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814387    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-systemd\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814424    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-registration-dir\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814426    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-bin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814446    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-host\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814449    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-systemd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814490    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-bin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814497    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-ovn\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814519    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-ovn\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814521    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-run-systemd\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814517    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-var-lib-kubelet\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814568    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-var-lib-kubelet\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814614    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-system-cni-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814643    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814661    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cnibin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814673    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-kubelet\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814682    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-system-cni-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814692    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-cnibin\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814690    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-node-log\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814721    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-kubelet\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814728    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-node-log\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814727    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814786    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cnibin\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814803    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-multus\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814806    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/858636bd-15aa-47e2-95ce-daaf7cffb8d0-cnibin\") pod \"multus-additional-cni-plugins-vx9mh\" (UID: \"858636bd-15aa-47e2-95ce-daaf7cffb8d0\") " pod="openshift-multus/multus-additional-cni-plugins-vx9mh"
Oct 31 00:29:52.814805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814816    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-multus-certs\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814838    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-var-lib-cni-multus\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814842    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-host-run-multus-certs\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814870    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-wtmp\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814891    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814917    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-socket-dir-parent\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814944    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-sys\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814980    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-multus-socket-dir-parent\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815004    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-node-exporter-wtmp\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.814984    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-host\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815021    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysctl-d\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815034    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-kubelet\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815042    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9d66308f-554a-4ea4-91ed-ffe42bbb0f3e-host\") pod \"node-ca-j9jsc\" (UID: \"9d66308f-554a-4ea4-91ed-ffe42bbb0f3e\") " pod="openshift-image-registry/node-ca-j9jsc"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815062    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-sys\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815068    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-sys\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815116    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-system-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815123    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-sys\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815072    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-kubelet\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815129    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-root\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815174    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02ae9325-803d-4acc-bafe-4349f1548558-root\") pod \"node-exporter-9cnsr\" (UID: \"02ae9325-803d-4acc-bafe-4349f1548558\") " pod="openshift-monitoring/node-exporter-9cnsr"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815181    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-kubelet-config\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815213    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-system-cni-dir\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815213    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a1760d4-777c-4853-bbbf-0a9afae8f45f-kubelet-config\") pod \"global-pull-secret-syncer-hrxqv\" (UID: \"2a1760d4-777c-4853-bbbf-0a9afae8f45f\") " pod="kube-system/global-pull-secret-syncer-hrxqv"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815235    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454cfe49-4029-4326-b0ce-f3fe9194d991-crio-socket\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815251    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815262    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/454cfe49-4029-4326-b0ce-f3fe9194d991-crio-socket\") pod \"insights-runtime-extractor-nm5ww\" (UID: \"454cfe49-4029-4326-b0ce-f3fe9194d991\") " pod="openshift-insights/insights-runtime-extractor-nm5ww"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815285    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-hostroot\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815264    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0aeb97e6-8dcd-4896-8a32-424681fcfc47-hostroot\") pod \"multus-jh76v\" (UID: \"0aeb97e6-8dcd-4896-8a32-424681fcfc47\") " pod="openshift-multus/multus-jh76v"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815291    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-sys-fs\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815319    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c36f1c1a-8072-409c-ba64-929fe596778b-hosts-file\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815337    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-run\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815394    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaec7db-74f2-49ac-a56b-198f78680fce-host-slash\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815402    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c36f1c1a-8072-409c-ba64-929fe596778b-hosts-file\") pod \"node-resolver-wjjv6\" (UID: \"c36f1c1a-8072-409c-ba64-929fe596778b\") " pod="openshift-dns/node-resolver-wjjv6"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815432    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-log-socket\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815450    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysconfig\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815464    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-run\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815488    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-log-socket\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815498    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815512    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815513    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaec7db-74f2-49ac-a56b-198f78680fce-host-slash\") pod \"iptables-alerter-cntts\" (UID: \"eaaec7db-74f2-49ac-a56b-198f78680fce\") " pod="openshift-network-operator/iptables-alerter-cntts"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815521    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7c477db4-252e-4236-9ddd-79307115124f-etc-sysconfig\") pod \"tuned-2rxnb\" (UID: \"7c477db4-252e-4236-9ddd-79307115124f\") " pod="openshift-cluster-node-tuning-operator/tuned-2rxnb"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815533    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-slash\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815568    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-slash\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815589    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/01df07af-c80e-4ae5-950d-87cc511cbda7-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxlg7\" (UID: \"01df07af-c80e-4ae5-950d-87cc511cbda7\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:29:52.816020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:52.815648    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d584253-d14a-48e9-bfe6-563acb954e1f-etc-selinux\") pod \"aws-ebs-csi-driver-node-5jpqn\" (UID: \"3d584253-d14a-48e9-bfe6-563acb954e1f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-5jpqn"
Oct 31 00:29:54.697858 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:54.697825    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:57.700706 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:57.700679    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kdkdw"
Oct 31 00:29:59.438579 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:29:59.438554    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gmcv9"
Oct 31 00:30:00.506680 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:00.506652    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:30:01.514693 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:01.514668    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:30:02.246685 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:02.246650    4888 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-131-97.us-west-2.compute.internal" event="NodeReady"
Oct 31 00:30:02.598843 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:02.598772    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:30:02.599187 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:02.599002    4888 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:30:06.826557 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:06.826526    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxlg7"
Oct 31 00:30:18.550958 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:30:18.550925    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:30:20.509428 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:20.509328    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mr44j"
Oct 31 00:30:26.228909 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.228876    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 00:30:26.232558 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.232537    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.284641 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.284610    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 00:30:26.284754 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.284650    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 00:30:26.284754 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.284677    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 00:30:26.284832 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.284790    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 00:30:26.304729 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.304700    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.304858 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.304751    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.304858 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.304834    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.304943 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.304869    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d3522d-c03d-466e-b816-25caa6905168-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.304975 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.304946    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d3522d-c03d-466e-b816-25caa6905168-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305011 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.304973    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305011 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.304992    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305011 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.305009    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-web-config\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305127 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.305045    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d7lq\" (UniqueName: \"kubernetes.io/projected/a9d3522d-c03d-466e-b816-25caa6905168-kube-api-access-4d7lq\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305127 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.305094    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d3522d-c03d-466e-b816-25caa6905168-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305224 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.305156    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d3522d-c03d-466e-b816-25caa6905168-config-out\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305224 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.305198    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.305286 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.305262    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d3522d-c03d-466e-b816-25caa6905168-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.330534 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.330513    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 00:30:26.330534 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.330513    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 00:30:26.330843 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.330820    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 00:30:26.330843 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.330832    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-skrl7\""
Oct 31 00:30:26.331582 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.331567    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 00:30:26.405784 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405757    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d3522d-c03d-466e-b816-25caa6905168-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.405784 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405788    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.405936 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405809    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.405936 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405826    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.405936 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405850    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d3522d-c03d-466e-b816-25caa6905168-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.405936 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405881    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d3522d-c03d-466e-b816-25caa6905168-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.405936 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405908    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.405936 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405933    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.406217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405958    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-web-config\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.406217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405976    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d7lq\" (UniqueName: \"kubernetes.io/projected/a9d3522d-c03d-466e-b816-25caa6905168-kube-api-access-4d7lq\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.406217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.405996    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d3522d-c03d-466e-b816-25caa6905168-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.406217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.406158    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d3522d-c03d-466e-b816-25caa6905168-config-out\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.406217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.406198    4888 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:30:26.406217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.406210    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.406541 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.406266    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a9d3522d-c03d-466e-b816-25caa6905168-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.406848 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.406787    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9d3522d-c03d-466e-b816-25caa6905168-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409068 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409023    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9d3522d-c03d-466e-b816-25caa6905168-config-out\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409181 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409125    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409381 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409339    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409381 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409371    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409744 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409715    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-config-volume\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409854 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409836    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409921 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409901    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9d3522d-c03d-466e-b816-25caa6905168-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.409956 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.409914    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-web-config\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.419683 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.419666    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 00:30:26.424992 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.424977    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 00:30:26.427719 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.427697    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9d3522d-c03d-466e-b816-25caa6905168-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.428932 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.428914    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a9d3522d-c03d-466e-b816-25caa6905168-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.465621 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.465599    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d7lq\" (UniqueName: \"kubernetes.io/projected/a9d3522d-c03d-466e-b816-25caa6905168-kube-api-access-4d7lq\") pod \"alertmanager-main-0\" (UID: \"a9d3522d-c03d-466e-b816-25caa6905168\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:26.549227 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:26.549158    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 00:30:27.013953 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:27.013923    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 00:30:27.017836 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:30:27.017812    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d3522d_c03d_466e_b816_25caa6905168.slice/crio-95dd7a04ec28e33b2257574752d27be132f255a5925c6b10dc0f5c6ed343d51b WatchSource:0}: Error finding container 95dd7a04ec28e33b2257574752d27be132f255a5925c6b10dc0f5c6ed343d51b: Status 404 returned error can't find the container with id 95dd7a04ec28e33b2257574752d27be132f255a5925c6b10dc0f5c6ed343d51b
Oct 31 00:30:27.019869 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:27.019849    4888 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:30:27.692860 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:27.692823    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerStarted","Data":"95dd7a04ec28e33b2257574752d27be132f255a5925c6b10dc0f5c6ed343d51b"}
Oct 31 00:30:29.701364 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:29.701319    4888 generic.go:356] "Generic (PLEG): container finished" podID="a9d3522d-c03d-466e-b816-25caa6905168" containerID="86edebec96e2e61cb6e2fe48aa05f7c029c58588ed725332724bacd68dbb41d1" exitCode=0
Oct 31 00:30:29.701745 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:29.701406    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerDied","Data":"86edebec96e2e61cb6e2fe48aa05f7c029c58588ed725332724bacd68dbb41d1"}
Oct 31 00:30:32.713176 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:32.713147    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerStarted","Data":"d4e4f5ab64eb28e4d640a5c2d320ca6739ebdc0caff3e10eda4e1b8c5fd90581"}
Oct 31 00:30:32.713556 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:32.713209    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerStarted","Data":"841e9b88c08d1fb30239236c9b25d40c8d2df0bdcae0853214b312add6610cd9"}
Oct 31 00:30:32.713556 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:32.713235    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerStarted","Data":"827977749a9a95ed1554fa8b8c7511d3a772297c24b0fbd6e01196238b9cafcc"}
Oct 31 00:30:32.713556 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:32.713256    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerStarted","Data":"2ab63809ce5e38f72c4a5c5f6c9b5fa4868db7fc122b1ef70f85e5c89257351c"}
Oct 31 00:30:32.713556 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:32.713279    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerStarted","Data":"479a79d37930cc3f52282ecf5f5fc33ea7a6a70039f3f8125b25681dd3092079"}
Oct 31 00:30:33.721929 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:33.721892    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a9d3522d-c03d-466e-b816-25caa6905168","Type":"ContainerStarted","Data":"379d0534a84f7d77425707c4dd0945ce139636523bceaaec19aef0842917c5eb"}
Oct 31 00:30:33.987568 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:30:33.987517    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.516048049 podStartE2EDuration="7.987502903s" totalImagesPullingTime="5.45743482s" totalInitContainerRuntime="14.020034ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:33.960891555 +0000 UTC m=+42.515778992" watchObservedRunningTime="2026-10-31 00:30:33.987502903 +0000 UTC m=+42.542390341"
Oct 31 00:30:48.551101 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:30:48.551067    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:31:18.550000 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:31:18.549961    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:31:48.550148 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:31:48.550097    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:32:18.550790 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:32:18.550702    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:32:38.032414 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.032383    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"]
Oct 31 00:32:38.035300 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.035285    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.133112 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.133079    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 00:32:38.133288 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.133117    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"98a21a31f7a8ae0212b9c22cc643f9202bfa37-6b211c7f-dockercfg-xtwrc\""
Oct 31 00:32:38.133288 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.133137    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 00:32:38.133635 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.133617    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"]
Oct 31 00:32:38.145424 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.145405    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cthms\" (UniqueName: \"kubernetes.io/projected/fcb3065b-b9e6-421e-83be-1481f80d2354-kube-api-access-cthms\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.145507 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.145431    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.145507 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.145457    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.246366 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.246326    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.246540 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.246422    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cthms\" (UniqueName: \"kubernetes.io/projected/fcb3065b-b9e6-421e-83be-1481f80d2354-kube-api-access-cthms\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.246540 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.246440    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.246781 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.246765    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.246826 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.246764    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.307395 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.307327    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cthms\" (UniqueName: \"kubernetes.io/projected/fcb3065b-b9e6-421e-83be-1481f80d2354-kube-api-access-cthms\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.350281 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.350253    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:32:38.720583 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:38.720553    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"]
Oct 31 00:32:38.724098 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:32:38.724067    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcb3065b_b9e6_421e_83be_1481f80d2354.slice/crio-2d126a54295f0082688efe31e382dc70bedc4e70d8f1a88cf42eea670f330f22 WatchSource:0}: Error finding container 2d126a54295f0082688efe31e382dc70bedc4e70d8f1a88cf42eea670f330f22: Status 404 returned error can't find the container with id 2d126a54295f0082688efe31e382dc70bedc4e70d8f1a88cf42eea670f330f22
Oct 31 00:32:39.089254 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:39.089175    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd" event={"ID":"fcb3065b-b9e6-421e-83be-1481f80d2354","Type":"ContainerStarted","Data":"2d126a54295f0082688efe31e382dc70bedc4e70d8f1a88cf42eea670f330f22"}
Oct 31 00:32:45.115874 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:45.115843    4888 generic.go:356] "Generic (PLEG): container finished" podID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerID="45229ada520d7686e770b0fc320823cf88020277b595690e962f663420d4e9df" exitCode=0
Oct 31 00:32:45.115874 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:45.115879    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd" event={"ID":"fcb3065b-b9e6-421e-83be-1481f80d2354","Type":"ContainerDied","Data":"45229ada520d7686e770b0fc320823cf88020277b595690e962f663420d4e9df"}
Oct 31 00:32:48.550987 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:32:48.550946    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:32:51.135598 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:51.135562    4888 generic.go:356] "Generic (PLEG): container finished" podID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerID="9d7ccff228e76595666fbc925aa5ce0828fc4551679bf141306ef3914dd77317" exitCode=0
Oct 31 00:32:51.136038 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:51.135606    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd" event={"ID":"fcb3065b-b9e6-421e-83be-1481f80d2354","Type":"ContainerDied","Data":"9d7ccff228e76595666fbc925aa5ce0828fc4551679bf141306ef3914dd77317"}
Oct 31 00:32:58.160189 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:58.160149    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd" event={"ID":"fcb3065b-b9e6-421e-83be-1481f80d2354","Type":"ContainerStarted","Data":"70078339a9186cb12acdb51136eb47967b62fef6f1b3453f398a35dbe795ef0f"}
Oct 31 00:32:58.275491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:58.275444    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd" podStartSLOduration=3.692934713 podStartE2EDuration="21.275431716s" totalImagesPullingTime="17.582497003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:58.275300901 +0000 UTC m=+186.830188338" watchObservedRunningTime="2026-10-31 00:32:58.275431716 +0000 UTC m=+186.830319148"
Oct 31 00:32:59.165808 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:59.165775    4888 generic.go:356] "Generic (PLEG): container finished" podID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerID="70078339a9186cb12acdb51136eb47967b62fef6f1b3453f398a35dbe795ef0f" exitCode=0
Oct 31 00:32:59.166170 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:32:59.165851    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd" event={"ID":"fcb3065b-b9e6-421e-83be-1481f80d2354","Type":"ContainerDied","Data":"70078339a9186cb12acdb51136eb47967b62fef6f1b3453f398a35dbe795ef0f"}
Oct 31 00:33:00.330478 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.330450    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:33:00.504183 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.504158    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-util\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-util\") pod \"fcb3065b-b9e6-421e-83be-1481f80d2354\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") "
Oct 31 00:33:00.504364 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.504195    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-bundle\") pod \"fcb3065b-b9e6-421e-83be-1481f80d2354\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") "
Oct 31 00:33:00.504364 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.504241    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcb3065b-b9e6-421e-83be-1481f80d2354-kube-api-access-cthms\" (UniqueName: \"kubernetes.io/projected/fcb3065b-b9e6-421e-83be-1481f80d2354-kube-api-access-cthms\") pod \"fcb3065b-b9e6-421e-83be-1481f80d2354\" (UID: \"fcb3065b-b9e6-421e-83be-1481f80d2354\") "
Oct 31 00:33:00.504852 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.504816    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-bundle" pod "fcb3065b-b9e6-421e-83be-1481f80d2354" (UID: "fcb3065b-b9e6-421e-83be-1481f80d2354"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:33:00.506480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.506458    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcb3065b-b9e6-421e-83be-1481f80d2354-kube-api-access-cthms" pod "fcb3065b-b9e6-421e-83be-1481f80d2354" (UID: "fcb3065b-b9e6-421e-83be-1481f80d2354"). InnerVolumeSpecName "kube-api-access-cthms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:00.508694 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.508672    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-util" pod "fcb3065b-b9e6-421e-83be-1481f80d2354" (UID: "fcb3065b-b9e6-421e-83be-1481f80d2354"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:33:00.604767 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.604738    4888 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-util\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:00.604767 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.604765    4888 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fcb3065b-b9e6-421e-83be-1481f80d2354-bundle\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:00.604904 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:00.604778    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cthms\" (UniqueName: \"kubernetes.io/projected/fcb3065b-b9e6-421e-83be-1481f80d2354-kube-api-access-cthms\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:33:01.174696 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:01.174649    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd" event={"ID":"fcb3065b-b9e6-421e-83be-1481f80d2354","Type":"ContainerDied","Data":"2d126a54295f0082688efe31e382dc70bedc4e70d8f1a88cf42eea670f330f22"}
Oct 31 00:33:01.174931 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:01.174710    4888 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d126a54295f0082688efe31e382dc70bedc4e70d8f1a88cf42eea670f330f22"
Oct 31 00:33:01.174931 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:01.174665    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820c8zpxd"
Oct 31 00:33:06.525628 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525596    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"]
Oct 31 00:33:06.526006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525815    4888 cpu_manager.go:414] "containerMap: removing container" podUID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerName="util"
Oct 31 00:33:06.526006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525824    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerName="util"
Oct 31 00:33:06.526006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525832    4888 cpu_manager.go:414] "containerMap: removing container" podUID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerName="extract"
Oct 31 00:33:06.526006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525837    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerName="extract"
Oct 31 00:33:06.526006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525859    4888 cpu_manager.go:414] "containerMap: removing container" podUID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerName="pull"
Oct 31 00:33:06.526006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525864    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerName="pull"
Oct 31 00:33:06.526006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.525900    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcb3065b-b9e6-421e-83be-1481f80d2354" containerName="extract"
Oct 31 00:33:06.527745 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.527730    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:06.544487 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.544460    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/27ed67bc-2e77-4ef4-940b-8717a26180d5-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-s8fql\" (UID: \"27ed67bc-2e77-4ef4-940b-8717a26180d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:06.544602 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.544500    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qbv\" (UniqueName: \"kubernetes.io/projected/27ed67bc-2e77-4ef4-940b-8717a26180d5-kube-api-access-52qbv\") pod \"custom-metrics-autoscaler-operator-c667576c8-s8fql\" (UID: \"27ed67bc-2e77-4ef4-940b-8717a26180d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:06.579008 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.578981    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-rtvch\""
Oct 31 00:33:06.579008 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.578992    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 31 00:33:06.624985 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.624957    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 31 00:33:06.624985 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.624964    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 31 00:33:06.625643 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.625627    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"]
Oct 31 00:33:06.644853 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.644827    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/27ed67bc-2e77-4ef4-940b-8717a26180d5-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-s8fql\" (UID: \"27ed67bc-2e77-4ef4-940b-8717a26180d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:06.644951 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.644865    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qbv\" (UniqueName: \"kubernetes.io/projected/27ed67bc-2e77-4ef4-940b-8717a26180d5-kube-api-access-52qbv\") pod \"custom-metrics-autoscaler-operator-c667576c8-s8fql\" (UID: \"27ed67bc-2e77-4ef4-940b-8717a26180d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:06.647172 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.647153    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/27ed67bc-2e77-4ef4-940b-8717a26180d5-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-s8fql\" (UID: \"27ed67bc-2e77-4ef4-940b-8717a26180d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:06.705445 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.705420    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qbv\" (UniqueName: \"kubernetes.io/projected/27ed67bc-2e77-4ef4-940b-8717a26180d5-kube-api-access-52qbv\") pod \"custom-metrics-autoscaler-operator-c667576c8-s8fql\" (UID: \"27ed67bc-2e77-4ef4-940b-8717a26180d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:06.838231 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:06.838160    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:07.260466 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:07.260436    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"]
Oct 31 00:33:07.262182 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:33:07.262156    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ed67bc_2e77_4ef4_940b_8717a26180d5.slice/crio-89721587249a384bea9ec615fa50bbb528a93c8ea5d3b81ea5892ef29b4c700b WatchSource:0}: Error finding container 89721587249a384bea9ec615fa50bbb528a93c8ea5d3b81ea5892ef29b4c700b: Status 404 returned error can't find the container with id 89721587249a384bea9ec615fa50bbb528a93c8ea5d3b81ea5892ef29b4c700b
Oct 31 00:33:08.197219 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:08.197182    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql" event={"ID":"27ed67bc-2e77-4ef4-940b-8717a26180d5","Type":"ContainerStarted","Data":"89721587249a384bea9ec615fa50bbb528a93c8ea5d3b81ea5892ef29b4c700b"}
Oct 31 00:33:15.221554 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:15.221516    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql" event={"ID":"27ed67bc-2e77-4ef4-940b-8717a26180d5","Type":"ContainerStarted","Data":"62bfb333e92867b38d9d22fd42da133f9e76450d65b3ccbbaaa951be5c06b0a5"}
Oct 31 00:33:15.221959 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:15.221748    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:15.431217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:15.431157    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql" podStartSLOduration=2.315057693 podStartE2EDuration="9.431138505s" totalImagesPullingTime="7.116080812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:15.383752526 +0000 UTC m=+203.938639963" watchObservedRunningTime="2026-10-31 00:33:15.431138505 +0000 UTC m=+203.986026016"
Oct 31 00:33:17.244486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.244456    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-7dc85965f-t246p"]
Oct 31 00:33:17.246297 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.246279    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.297289 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.297265    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Oct 31 00:33:17.321780 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.321751    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-cabundle0\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.321903 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.321788    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-certificates\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.321903 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.321863    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw928\" (UniqueName: \"kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-kube-api-access-bw928\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.343428 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.343409    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 31 00:33:17.343491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.343452    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-qppcs\""
Oct 31 00:33:17.344131 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.344113    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-t246p"]
Oct 31 00:33:17.423044 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.423016    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-cabundle0\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.423162 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.423050    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-certificates\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.423162 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.423093    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw928\" (UniqueName: \"kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-kube-api-access-bw928\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.423162 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:17.423137    4888 configmap.go:193] Couldn't get configMap openshift-keda/keda-ocp-cabundle: configmap "keda-ocp-cabundle" not found
Oct 31 00:33:17.423276 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:17.423174    4888 secret.go:281] references non-existent secret key: ca.crt
Oct 31 00:33:17.423276 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:17.423190    4888 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Oct 31 00:33:17.423276 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:17.423198    4888 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-t246p: references non-existent secret key: ca.crt
Oct 31 00:33:17.423276 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:17.423201    4888 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-cabundle0 podName:8d1e82ea-8534-4c42-895c-4a1fc28f61d1 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:17.923180023 +0000 UTC m=+206.478067454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cabundle0" (UniqueName: "kubernetes.io/configmap/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-cabundle0") pod "keda-operator-7dc85965f-t246p" (UID: "8d1e82ea-8534-4c42-895c-4a1fc28f61d1") : configmap "keda-ocp-cabundle" not found
Oct 31 00:33:17.423276 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:17.423239    4888 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-certificates podName:8d1e82ea-8534-4c42-895c-4a1fc28f61d1 nodeName:}" failed. No retries permitted until 2026-10-31 00:33:17.923227364 +0000 UTC m=+206.478114795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-certificates") pod "keda-operator-7dc85965f-t246p" (UID: "8d1e82ea-8534-4c42-895c-4a1fc28f61d1") : references non-existent secret key: ca.crt
Oct 31 00:33:17.485122 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.485098    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw928\" (UniqueName: \"kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-kube-api-access-bw928\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.926516 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.926479    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-cabundle0\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.926516 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.926524    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-certificates\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.927188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.927168    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-cabundle0\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:17.929180 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:17.929161    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8d1e82ea-8534-4c42-895c-4a1fc28f61d1-certificates\") pod \"keda-operator-7dc85965f-t246p\" (UID: \"8d1e82ea-8534-4c42-895c-4a1fc28f61d1\") " pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:18.156303 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.156272    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:18.372120 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.372084    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"]
Oct 31 00:33:18.374127 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.374113    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.427588 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.427568    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 31 00:33:18.429834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.429817    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.429904 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.429858    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8n7\" (UniqueName: \"kubernetes.io/projected/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-kube-api-access-jq8n7\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.429904 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.429882    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.431954 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.431930    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"]
Oct 31 00:33:18.530661 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.530592    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8n7\" (UniqueName: \"kubernetes.io/projected/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-kube-api-access-jq8n7\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.530661 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.530636    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.530891 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.530825    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.531143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.531127    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.533300 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.533278    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.539601 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.539580    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-t246p"]
Oct 31 00:33:18.542901 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:33:18.542879    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1e82ea_8534_4c42_895c_4a1fc28f61d1.slice/crio-f6945f55dab68d218facade011f18ea673c6b67e6347dc1178e981996a552648 WatchSource:0}: Error finding container f6945f55dab68d218facade011f18ea673c6b67e6347dc1178e981996a552648: Status 404 returned error can't find the container with id f6945f55dab68d218facade011f18ea673c6b67e6347dc1178e981996a552648
Oct 31 00:33:18.552645 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:18.552616    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:33:18.591397 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.591370    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8n7\" (UniqueName: \"kubernetes.io/projected/ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd-kube-api-access-jq8n7\") pod \"keda-metrics-apiserver-86bd856cfb-tfdq5\" (UID: \"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.685495 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.685467    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:18.866148 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.866083    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-l7zhv"]
Oct 31 00:33:18.868398 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.868379    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:18.919514 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.919496    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 31 00:33:18.925597 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.925294    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-l7zhv"]
Oct 31 00:33:18.933549 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.933528    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbqf\" (UniqueName: \"kubernetes.io/projected/fff556d6-fb0d-467d-90f8-ea2f50c53613-kube-api-access-6dbqf\") pod \"keda-admission-7b7c985c58-l7zhv\" (UID: \"fff556d6-fb0d-467d-90f8-ea2f50c53613\") " pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:18.933653 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:18.933593    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fff556d6-fb0d-467d-90f8-ea2f50c53613-certificates\") pod \"keda-admission-7b7c985c58-l7zhv\" (UID: \"fff556d6-fb0d-467d-90f8-ea2f50c53613\") " pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:19.034409 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.034383    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fff556d6-fb0d-467d-90f8-ea2f50c53613-certificates\") pod \"keda-admission-7b7c985c58-l7zhv\" (UID: \"fff556d6-fb0d-467d-90f8-ea2f50c53613\") " pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:19.034545 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.034421    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbqf\" (UniqueName: \"kubernetes.io/projected/fff556d6-fb0d-467d-90f8-ea2f50c53613-kube-api-access-6dbqf\") pod \"keda-admission-7b7c985c58-l7zhv\" (UID: \"fff556d6-fb0d-467d-90f8-ea2f50c53613\") " pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:19.038376 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.037924    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fff556d6-fb0d-467d-90f8-ea2f50c53613-certificates\") pod \"keda-admission-7b7c985c58-l7zhv\" (UID: \"fff556d6-fb0d-467d-90f8-ea2f50c53613\") " pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:19.047141 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.047120    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"]
Oct 31 00:33:19.050231 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:33:19.050205    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6d47a3_5ee6_404a_a4c3_19f9a6d2a2fd.slice/crio-46d9517756a6ca854e61341d695465471963ee7937695b07a162691125280e3f WatchSource:0}: Error finding container 46d9517756a6ca854e61341d695465471963ee7937695b07a162691125280e3f: Status 404 returned error can't find the container with id 46d9517756a6ca854e61341d695465471963ee7937695b07a162691125280e3f
Oct 31 00:33:19.095020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.094993    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbqf\" (UniqueName: \"kubernetes.io/projected/fff556d6-fb0d-467d-90f8-ea2f50c53613-kube-api-access-6dbqf\") pod \"keda-admission-7b7c985c58-l7zhv\" (UID: \"fff556d6-fb0d-467d-90f8-ea2f50c53613\") " pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:19.179763 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.179706    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:19.237250 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.237222    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5" event={"ID":"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd","Type":"ContainerStarted","Data":"46d9517756a6ca854e61341d695465471963ee7937695b07a162691125280e3f"}
Oct 31 00:33:19.238394 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.238371    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-t246p" event={"ID":"8d1e82ea-8534-4c42-895c-4a1fc28f61d1","Type":"ContainerStarted","Data":"f6945f55dab68d218facade011f18ea673c6b67e6347dc1178e981996a552648"}
Oct 31 00:33:19.541596 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:19.541564    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-l7zhv"]
Oct 31 00:33:19.545651 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:33:19.545624    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff556d6_fb0d_467d_90f8_ea2f50c53613.slice/crio-4be0cec6e1c1130f136151361b6570d682626351c2b576d27e755356d89f4350 WatchSource:0}: Error finding container 4be0cec6e1c1130f136151361b6570d682626351c2b576d27e755356d89f4350: Status 404 returned error can't find the container with id 4be0cec6e1c1130f136151361b6570d682626351c2b576d27e755356d89f4350
Oct 31 00:33:20.243118 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:20.243043    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-l7zhv" event={"ID":"fff556d6-fb0d-467d-90f8-ea2f50c53613","Type":"ContainerStarted","Data":"4be0cec6e1c1130f136151361b6570d682626351c2b576d27e755356d89f4350"}
Oct 31 00:33:23.256674 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:23.256637    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-l7zhv" event={"ID":"fff556d6-fb0d-467d-90f8-ea2f50c53613","Type":"ContainerStarted","Data":"becb1c890c38bcff2f3c44100082bdcc02386de52b97d5516b6c14d7e57e6ca4"}
Oct 31 00:33:23.257131 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:23.256746    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:23.369784 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:23.369721    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-l7zhv" podStartSLOduration=2.329773818 podStartE2EDuration="5.369699384s" totalImagesPullingTime="3.039925566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:23.368792026 +0000 UTC m=+211.923679476" watchObservedRunningTime="2026-10-31 00:33:23.369699384 +0000 UTC m=+211.924586823"
Oct 31 00:33:24.270285 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:24.270187    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-t246p" event={"ID":"8d1e82ea-8534-4c42-895c-4a1fc28f61d1","Type":"ContainerStarted","Data":"a031ed021eed17f4d5e8a7c2fba9bb27c6d34d06b2939169930e5c69262b5e2d"}
Oct 31 00:33:24.386284 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:24.386230    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-operator-7dc85965f-t246p" podStartSLOduration=1.79202736 podStartE2EDuration="7.386216265s" totalImagesPullingTime="5.594188905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:24.385428034 +0000 UTC m=+212.940315472" watchObservedRunningTime="2026-10-31 00:33:24.386216265 +0000 UTC m=+212.941103701"
Oct 31 00:33:25.275082 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:25.275052    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5" event={"ID":"ec6d47a3-5ee6-404a-a4c3-19f9a6d2a2fd","Type":"ContainerStarted","Data":"ade04358257d5ac4f99bfe077ded0ca3eee45722f9f506ac7392188e120c0cd5"}
Oct 31 00:33:25.275505 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:25.275324    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:25.275505 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:25.275391    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:25.388955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:25.388909    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5" podStartSLOduration=1.900080885 podStartE2EDuration="7.388894448s" totalImagesPullingTime="5.488813563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:25.388175901 +0000 UTC m=+213.943063350" watchObservedRunningTime="2026-10-31 00:33:25.388894448 +0000 UTC m=+213.943781915"
Oct 31 00:33:36.227159 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:36.227131    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-s8fql"
Oct 31 00:33:36.284901 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:36.284873    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-tfdq5"
Oct 31 00:33:44.272273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:44.272244    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-l7zhv"
Oct 31 00:33:46.281085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:33:46.281055    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-7dc85965f-t246p"
Oct 31 00:33:48.550829 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:33:48.550791    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:34:10.974412 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:10.974378    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"]
Oct 31 00:34:10.976451 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:10.976435    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.074118 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.074092    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 00:34:11.074118 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.074093    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-5g492\""
Oct 31 00:34:11.074118 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.074093    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 00:34:11.075084 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.075066    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"]
Oct 31 00:34:11.103983 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.103954    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmvlb\" (UniqueName: \"kubernetes.io/projected/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-kube-api-access-zmvlb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.104116 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.103987    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.104116 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.104016    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.204791 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.204748    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.204978 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.204833    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmvlb\" (UniqueName: \"kubernetes.io/projected/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-kube-api-access-zmvlb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.204978 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.204868    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.205156 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.205136    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.205220 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.205191    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.265999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.265933    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmvlb\" (UniqueName: \"kubernetes.io/projected/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-kube-api-access-zmvlb\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.287735 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.287708    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:11.650867 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:11.650792    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"]
Oct 31 00:34:11.653921 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:34:11.653895    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab90fa34_5b9c_4f19_b1db_30b2b84a027d.slice/crio-256c80f54f3dd6d7338f7f4915f2104fc8910b2d652e41e4fb27fcfe4b72f55c WatchSource:0}: Error finding container 256c80f54f3dd6d7338f7f4915f2104fc8910b2d652e41e4fb27fcfe4b72f55c: Status 404 returned error can't find the container with id 256c80f54f3dd6d7338f7f4915f2104fc8910b2d652e41e4fb27fcfe4b72f55c
Oct 31 00:34:12.437417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:12.437384    4888 generic.go:356] "Generic (PLEG): container finished" podID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerID="b832ddffcd401deb4f74084cb1e4f449e4445c9fc291b9ba2b1dd88184e56756" exitCode=0
Oct 31 00:34:12.437886 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:12.437436    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s" event={"ID":"ab90fa34-5b9c-4f19-b1db-30b2b84a027d","Type":"ContainerDied","Data":"b832ddffcd401deb4f74084cb1e4f449e4445c9fc291b9ba2b1dd88184e56756"}
Oct 31 00:34:12.437886 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:12.437470    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s" event={"ID":"ab90fa34-5b9c-4f19-b1db-30b2b84a027d","Type":"ContainerStarted","Data":"256c80f54f3dd6d7338f7f4915f2104fc8910b2d652e41e4fb27fcfe4b72f55c"}
Oct 31 00:34:16.453095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:16.453064    4888 generic.go:356] "Generic (PLEG): container finished" podID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerID="bb431bef1b57eccceb8a20f6cffbb5383009528d844c03c18dd81ffbff7c6a53" exitCode=0
Oct 31 00:34:16.453491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:16.453103    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s" event={"ID":"ab90fa34-5b9c-4f19-b1db-30b2b84a027d","Type":"ContainerDied","Data":"bb431bef1b57eccceb8a20f6cffbb5383009528d844c03c18dd81ffbff7c6a53"}
Oct 31 00:34:17.458092 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:17.458059    4888 generic.go:356] "Generic (PLEG): container finished" podID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerID="10f324497c2730108da1dc797e5300b63f2210b83cf5a2796b679b7bb93e98b0" exitCode=0
Oct 31 00:34:17.458489 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:17.458153    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s" event={"ID":"ab90fa34-5b9c-4f19-b1db-30b2b84a027d","Type":"ContainerDied","Data":"10f324497c2730108da1dc797e5300b63f2210b83cf5a2796b679b7bb93e98b0"}
Oct 31 00:34:18.552180 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:34:18.552142    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:34:18.583510 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.583491    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:18.763608 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.763536    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-kube-api-access-zmvlb\" (UniqueName: \"kubernetes.io/projected/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-kube-api-access-zmvlb\") pod \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") "
Oct 31 00:34:18.763608 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.763577    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-bundle\") pod \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") "
Oct 31 00:34:18.763833 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.763610    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-util\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-util\") pod \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\" (UID: \"ab90fa34-5b9c-4f19-b1db-30b2b84a027d\") "
Oct 31 00:34:18.764216 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.764189    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-bundle" pod "ab90fa34-5b9c-4f19-b1db-30b2b84a027d" (UID: "ab90fa34-5b9c-4f19-b1db-30b2b84a027d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:18.765646 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.765624    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-kube-api-access-zmvlb" pod "ab90fa34-5b9c-4f19-b1db-30b2b84a027d" (UID: "ab90fa34-5b9c-4f19-b1db-30b2b84a027d"). InnerVolumeSpecName "kube-api-access-zmvlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:18.767759 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.767735    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-util" pod "ab90fa34-5b9c-4f19-b1db-30b2b84a027d" (UID: "ab90fa34-5b9c-4f19-b1db-30b2b84a027d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:18.864085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.864063    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmvlb\" (UniqueName: \"kubernetes.io/projected/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-kube-api-access-zmvlb\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:18.864085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.864084    4888 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-bundle\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:18.864203 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:18.864093    4888 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ab90fa34-5b9c-4f19-b1db-30b2b84a027d-util\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:19.466637 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:19.466600    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s" event={"ID":"ab90fa34-5b9c-4f19-b1db-30b2b84a027d","Type":"ContainerDied","Data":"256c80f54f3dd6d7338f7f4915f2104fc8910b2d652e41e4fb27fcfe4b72f55c"}
Oct 31 00:34:19.466805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:19.466659    4888 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="256c80f54f3dd6d7338f7f4915f2104fc8910b2d652e41e4fb27fcfe4b72f55c"
Oct 31 00:34:19.466805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:19.466614    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cb7z4s"
Oct 31 00:34:24.218261 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218232    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"]
Oct 31 00:34:24.218740 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218499    4888 cpu_manager.go:414] "containerMap: removing container" podUID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerName="util"
Oct 31 00:34:24.218740 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218509    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerName="util"
Oct 31 00:34:24.218740 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218517    4888 cpu_manager.go:414] "containerMap: removing container" podUID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerName="extract"
Oct 31 00:34:24.218740 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218523    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerName="extract"
Oct 31 00:34:24.218740 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218532    4888 cpu_manager.go:414] "containerMap: removing container" podUID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerName="pull"
Oct 31 00:34:24.218740 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218537    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerName="pull"
Oct 31 00:34:24.218740 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.218598    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab90fa34-5b9c-4f19-b1db-30b2b84a027d" containerName="extract"
Oct 31 00:34:24.221245 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.221230    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:24.283213 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.283187    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-28r5b\""
Oct 31 00:34:24.321973 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.321940    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:34:24.322114 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.322003    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Oct 31 00:34:24.322906 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.322888    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"]
Oct 31 00:34:24.396474 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.396439    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01a6a452-8b71-4761-bbf5-3fdb9f5675a3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7sghq\" (UID: \"01a6a452-8b71-4761-bbf5-3fdb9f5675a3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:24.396628 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.396484    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcdt9\" (UniqueName: \"kubernetes.io/projected/01a6a452-8b71-4761-bbf5-3fdb9f5675a3-kube-api-access-pcdt9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7sghq\" (UID: \"01a6a452-8b71-4761-bbf5-3fdb9f5675a3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:24.497852 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.497794    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01a6a452-8b71-4761-bbf5-3fdb9f5675a3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7sghq\" (UID: \"01a6a452-8b71-4761-bbf5-3fdb9f5675a3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:24.497852 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.497825    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcdt9\" (UniqueName: \"kubernetes.io/projected/01a6a452-8b71-4761-bbf5-3fdb9f5675a3-kube-api-access-pcdt9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7sghq\" (UID: \"01a6a452-8b71-4761-bbf5-3fdb9f5675a3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:24.498151 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.498132    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01a6a452-8b71-4761-bbf5-3fdb9f5675a3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7sghq\" (UID: \"01a6a452-8b71-4761-bbf5-3fdb9f5675a3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:24.563820 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.563793    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcdt9\" (UniqueName: \"kubernetes.io/projected/01a6a452-8b71-4761-bbf5-3fdb9f5675a3-kube-api-access-pcdt9\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-7sghq\" (UID: \"01a6a452-8b71-4761-bbf5-3fdb9f5675a3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:24.831366 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:24.831278    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"
Oct 31 00:34:25.306779 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:25.306602    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq"]
Oct 31 00:34:25.309461 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:34:25.309435    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a6a452_8b71_4761_bbf5_3fdb9f5675a3.slice/crio-b5218c1a424fcf92841275a84ece2da97506d766ed4ffa028acd2680fb03c526 WatchSource:0}: Error finding container b5218c1a424fcf92841275a84ece2da97506d766ed4ffa028acd2680fb03c526: Status 404 returned error can't find the container with id b5218c1a424fcf92841275a84ece2da97506d766ed4ffa028acd2680fb03c526
Oct 31 00:34:25.492333 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:25.492297    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq" event={"ID":"01a6a452-8b71-4761-bbf5-3fdb9f5675a3","Type":"ContainerStarted","Data":"b5218c1a424fcf92841275a84ece2da97506d766ed4ffa028acd2680fb03c526"}
Oct 31 00:34:28.504302 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:28.504272    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq" event={"ID":"01a6a452-8b71-4761-bbf5-3fdb9f5675a3","Type":"ContainerStarted","Data":"a765dca17f574dbc1a43373f30bfadb3c2c3b8f85ce8e008b41f3450d0359f63"}
Oct 31 00:34:28.621093 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:28.621049    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-7sghq" podStartSLOduration=1.944240364 podStartE2EDuration="4.621035231s" totalImagesPullingTime="2.676794867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:28.62001417 +0000 UTC m=+277.174901627" watchObservedRunningTime="2026-10-31 00:34:28.621035231 +0000 UTC m=+277.175922668"
Oct 31 00:34:32.501561 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.501528    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"]
Oct 31 00:34:32.504833 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.504818    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:32.548245 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.548223    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3198ae19-1821-4342-9a33-fff5e30631c2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cvztz\" (UID: \"3198ae19-1821-4342-9a33-fff5e30631c2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:32.548390 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.548261    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmc68\" (UniqueName: \"kubernetes.io/projected/3198ae19-1821-4342-9a33-fff5e30631c2-kube-api-access-vmc68\") pod \"cert-manager-cainjector-5c57bbd4fc-cvztz\" (UID: \"3198ae19-1821-4342-9a33-fff5e30631c2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:32.564088 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.564063    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"]
Oct 31 00:34:32.605878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.605855    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 00:34:32.605878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.605864    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 00:34:32.605878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.605874    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-z9kss\""
Oct 31 00:34:32.649325 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.649295    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3198ae19-1821-4342-9a33-fff5e30631c2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cvztz\" (UID: \"3198ae19-1821-4342-9a33-fff5e30631c2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:32.649476 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.649337    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmc68\" (UniqueName: \"kubernetes.io/projected/3198ae19-1821-4342-9a33-fff5e30631c2-kube-api-access-vmc68\") pod \"cert-manager-cainjector-5c57bbd4fc-cvztz\" (UID: \"3198ae19-1821-4342-9a33-fff5e30631c2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:32.709220 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.709189    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3198ae19-1821-4342-9a33-fff5e30631c2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-cvztz\" (UID: \"3198ae19-1821-4342-9a33-fff5e30631c2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:32.709415 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.709395    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmc68\" (UniqueName: \"kubernetes.io/projected/3198ae19-1821-4342-9a33-fff5e30631c2-kube-api-access-vmc68\") pod \"cert-manager-cainjector-5c57bbd4fc-cvztz\" (UID: \"3198ae19-1821-4342-9a33-fff5e30631c2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:32.833526 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:32.833444    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"
Oct 31 00:34:33.199549 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:33.199522    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz"]
Oct 31 00:34:33.202309 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:34:33.202278    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3198ae19_1821_4342_9a33_fff5e30631c2.slice/crio-e63cee9bd63954390c427301316e2834f892b7e1d414b62c4886cdbd941c391b WatchSource:0}: Error finding container e63cee9bd63954390c427301316e2834f892b7e1d414b62c4886cdbd941c391b: Status 404 returned error can't find the container with id e63cee9bd63954390c427301316e2834f892b7e1d414b62c4886cdbd941c391b
Oct 31 00:34:33.524013 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:33.523940    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz" event={"ID":"3198ae19-1821-4342-9a33-fff5e30631c2","Type":"ContainerStarted","Data":"e63cee9bd63954390c427301316e2834f892b7e1d414b62c4886cdbd941c391b"}
Oct 31 00:34:34.212254 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.212223    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"]
Oct 31 00:34:34.216909 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.216886    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.260595 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.260567    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.260595 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.260592    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.260768 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.260635    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swl7p\" (UniqueName: \"kubernetes.io/projected/c60526f7-ff1a-448c-879b-516189f338df-kube-api-access-swl7p\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.314004 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.313981    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 00:34:34.314138 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.314019    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 00:34:34.314138 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.314120    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-lk7pz\""
Oct 31 00:34:34.314956 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.314941    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"]
Oct 31 00:34:34.361099 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.361071    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.361099 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.361096    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.361293 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.361115    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swl7p\" (UniqueName: \"kubernetes.io/projected/c60526f7-ff1a-448c-879b-516189f338df-kube-api-access-swl7p\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.361490 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.361467    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.361560 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.361505    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.421637 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.421603    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swl7p\" (UniqueName: \"kubernetes.io/projected/c60526f7-ff1a-448c-879b-516189f338df-kube-api-access-swl7p\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.533260 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.533191    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:34.896438 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:34.896341    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"]
Oct 31 00:34:34.898864 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:34:34.898839    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60526f7_ff1a_448c_879b_516189f338df.slice/crio-dfd5e2fe487dda703f9bbad5b9fd1fc2a69a4cf19a7fc1cbfa4f4d3d25e7e366 WatchSource:0}: Error finding container dfd5e2fe487dda703f9bbad5b9fd1fc2a69a4cf19a7fc1cbfa4f4d3d25e7e366: Status 404 returned error can't find the container with id dfd5e2fe487dda703f9bbad5b9fd1fc2a69a4cf19a7fc1cbfa4f4d3d25e7e366
Oct 31 00:34:35.533681 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:35.533646    4888 generic.go:356] "Generic (PLEG): container finished" podID="c60526f7-ff1a-448c-879b-516189f338df" containerID="9e89954fc15882b0ad6a78940383a9d63078e6d25b603a62baa3f90bd52dcb11" exitCode=0
Oct 31 00:34:35.534109 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:35.533693    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79" event={"ID":"c60526f7-ff1a-448c-879b-516189f338df","Type":"ContainerDied","Data":"9e89954fc15882b0ad6a78940383a9d63078e6d25b603a62baa3f90bd52dcb11"}
Oct 31 00:34:35.534109 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:35.533737    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79" event={"ID":"c60526f7-ff1a-448c-879b-516189f338df","Type":"ContainerStarted","Data":"dfd5e2fe487dda703f9bbad5b9fd1fc2a69a4cf19a7fc1cbfa4f4d3d25e7e366"}
Oct 31 00:34:37.544178 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:37.544141    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz" event={"ID":"3198ae19-1821-4342-9a33-fff5e30631c2","Type":"ContainerStarted","Data":"a99023fda24787fdb4651d006c33734d3eb37be2ee506e5536e52b605b12ce12"}
Oct 31 00:34:37.655326 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:37.655269    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-cvztz" podStartSLOduration=1.761616642 podStartE2EDuration="5.655253671s" totalImagesPullingTime="3.893637029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:37.655077195 +0000 UTC m=+286.209964633" watchObservedRunningTime="2026-10-31 00:34:37.655253671 +0000 UTC m=+286.210141110"
Oct 31 00:34:39.554154 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.554120    4888 generic.go:356] "Generic (PLEG): container finished" podID="c60526f7-ff1a-448c-879b-516189f338df" containerID="fe53c18a5708a5087ee6082e9dfbb281e957ea2c5789b40af499669c4491422f" exitCode=0
Oct 31 00:34:39.554557 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.554198    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79" event={"ID":"c60526f7-ff1a-448c-879b-516189f338df","Type":"ContainerDied","Data":"fe53c18a5708a5087ee6082e9dfbb281e957ea2c5789b40af499669c4491422f"}
Oct 31 00:34:39.596669 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.596646    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbxpj"]
Oct 31 00:34:39.601461 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.601441    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:39.656018 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.655987    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbxpj"]
Oct 31 00:34:39.657184 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.657171    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ln5nr\""
Oct 31 00:34:39.702264 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.702236    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49eeb9c6-033b-4ec0-ba1f-b39e872664e7-bound-sa-token\") pod \"cert-manager-597b95cb84-rbxpj\" (UID: \"49eeb9c6-033b-4ec0-ba1f-b39e872664e7\") " pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:39.702440 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.702269    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdv5r\" (UniqueName: \"kubernetes.io/projected/49eeb9c6-033b-4ec0-ba1f-b39e872664e7-kube-api-access-mdv5r\") pod \"cert-manager-597b95cb84-rbxpj\" (UID: \"49eeb9c6-033b-4ec0-ba1f-b39e872664e7\") " pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:39.803043 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.803006    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49eeb9c6-033b-4ec0-ba1f-b39e872664e7-bound-sa-token\") pod \"cert-manager-597b95cb84-rbxpj\" (UID: \"49eeb9c6-033b-4ec0-ba1f-b39e872664e7\") " pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:39.803043 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.803045    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdv5r\" (UniqueName: \"kubernetes.io/projected/49eeb9c6-033b-4ec0-ba1f-b39e872664e7-kube-api-access-mdv5r\") pod \"cert-manager-597b95cb84-rbxpj\" (UID: \"49eeb9c6-033b-4ec0-ba1f-b39e872664e7\") " pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:39.863229 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.863154    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49eeb9c6-033b-4ec0-ba1f-b39e872664e7-bound-sa-token\") pod \"cert-manager-597b95cb84-rbxpj\" (UID: \"49eeb9c6-033b-4ec0-ba1f-b39e872664e7\") " pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:39.863399 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.863342    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdv5r\" (UniqueName: \"kubernetes.io/projected/49eeb9c6-033b-4ec0-ba1f-b39e872664e7-kube-api-access-mdv5r\") pod \"cert-manager-597b95cb84-rbxpj\" (UID: \"49eeb9c6-033b-4ec0-ba1f-b39e872664e7\") " pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:39.912399 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:39.912362    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rbxpj"
Oct 31 00:34:40.278508 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:40.278478    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rbxpj"]
Oct 31 00:34:40.281539 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:34:40.281515    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49eeb9c6_033b_4ec0_ba1f_b39e872664e7.slice/crio-74ee1c776d2115d2ff9426d6f1dffe8f2760d677234f9860691f851dfa0a9915 WatchSource:0}: Error finding container 74ee1c776d2115d2ff9426d6f1dffe8f2760d677234f9860691f851dfa0a9915: Status 404 returned error can't find the container with id 74ee1c776d2115d2ff9426d6f1dffe8f2760d677234f9860691f851dfa0a9915
Oct 31 00:34:40.559566 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:40.559483    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rbxpj" event={"ID":"49eeb9c6-033b-4ec0-ba1f-b39e872664e7","Type":"ContainerStarted","Data":"46e15f3896867e211c3b42211949215602a8257b7d5da84fa6d7f7f249b06d54"}
Oct 31 00:34:40.559566 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:40.559538    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rbxpj" event={"ID":"49eeb9c6-033b-4ec0-ba1f-b39e872664e7","Type":"ContainerStarted","Data":"74ee1c776d2115d2ff9426d6f1dffe8f2760d677234f9860691f851dfa0a9915"}
Oct 31 00:34:40.561622 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:40.561598    4888 generic.go:356] "Generic (PLEG): container finished" podID="c60526f7-ff1a-448c-879b-516189f338df" containerID="64914a549b2762b5c4f24860f3ee264e08bb1af27c2832a2231f62644adf4d72" exitCode=0
Oct 31 00:34:40.561728 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:40.561658    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79" event={"ID":"c60526f7-ff1a-448c-879b-516189f338df","Type":"ContainerDied","Data":"64914a549b2762b5c4f24860f3ee264e08bb1af27c2832a2231f62644adf4d72"}
Oct 31 00:34:40.671730 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:40.671688    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rbxpj" podStartSLOduration=1.671674793 podStartE2EDuration="1.671674793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:40.670681284 +0000 UTC m=+289.225568718" watchObservedRunningTime="2026-10-31 00:34:40.671674793 +0000 UTC m=+289.226562231"
Oct 31 00:34:41.681199 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.681174    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:41.716397 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.716373    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-bundle\") pod \"c60526f7-ff1a-448c-879b-516189f338df\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") "
Oct 31 00:34:41.716537 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.716434    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60526f7-ff1a-448c-879b-516189f338df-kube-api-access-swl7p\" (UniqueName: \"kubernetes.io/projected/c60526f7-ff1a-448c-879b-516189f338df-kube-api-access-swl7p\") pod \"c60526f7-ff1a-448c-879b-516189f338df\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") "
Oct 31 00:34:41.716537 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.716458    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-util\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-util\") pod \"c60526f7-ff1a-448c-879b-516189f338df\" (UID: \"c60526f7-ff1a-448c-879b-516189f338df\") "
Oct 31 00:34:41.716783 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.716758    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-bundle" pod "c60526f7-ff1a-448c-879b-516189f338df" (UID: "c60526f7-ff1a-448c-879b-516189f338df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:41.718609 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.718580    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60526f7-ff1a-448c-879b-516189f338df-kube-api-access-swl7p" pod "c60526f7-ff1a-448c-879b-516189f338df" (UID: "c60526f7-ff1a-448c-879b-516189f338df"). InnerVolumeSpecName "kube-api-access-swl7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:41.720354 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.720327    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-util" pod "c60526f7-ff1a-448c-879b-516189f338df" (UID: "c60526f7-ff1a-448c-879b-516189f338df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:41.817703 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.817624    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swl7p\" (UniqueName: \"kubernetes.io/projected/c60526f7-ff1a-448c-879b-516189f338df-kube-api-access-swl7p\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:41.817703 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.817649    4888 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-util\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:41.817703 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:41.817659    4888 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c60526f7-ff1a-448c-879b-516189f338df-bundle\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:34:42.570774 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:42.570743    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79" event={"ID":"c60526f7-ff1a-448c-879b-516189f338df","Type":"ContainerDied","Data":"dfd5e2fe487dda703f9bbad5b9fd1fc2a69a4cf19a7fc1cbfa4f4d3d25e7e366"}
Oct 31 00:34:42.570951 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:42.570794    4888 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfd5e2fe487dda703f9bbad5b9fd1fc2a69a4cf19a7fc1cbfa4f4d3d25e7e366"
Oct 31 00:34:42.570951 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:42.570752    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855cn79"
Oct 31 00:34:48.551191 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:34:48.551155    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:34:51.575164 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:51.575088    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:34:51.577228 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:34:51.577208    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:35:02.943762 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.943728    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"]
Oct 31 00:35:02.944143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.943997    4888 cpu_manager.go:414] "containerMap: removing container" podUID="c60526f7-ff1a-448c-879b-516189f338df" containerName="util"
Oct 31 00:35:02.944143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.944006    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60526f7-ff1a-448c-879b-516189f338df" containerName="util"
Oct 31 00:35:02.944143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.944022    4888 cpu_manager.go:414] "containerMap: removing container" podUID="c60526f7-ff1a-448c-879b-516189f338df" containerName="extract"
Oct 31 00:35:02.944143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.944027    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60526f7-ff1a-448c-879b-516189f338df" containerName="extract"
Oct 31 00:35:02.944143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.944033    4888 cpu_manager.go:414] "containerMap: removing container" podUID="c60526f7-ff1a-448c-879b-516189f338df" containerName="pull"
Oct 31 00:35:02.944143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.944038    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60526f7-ff1a-448c-879b-516189f338df" containerName="pull"
Oct 31 00:35:02.944143 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.944081    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60526f7-ff1a-448c-879b-516189f338df" containerName="extract"
Oct 31 00:35:02.946825 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.946809    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:02.998601 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.998576    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 00:35:02.998601 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:02.998586    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6b2wx\""
Oct 31 00:35:03.045279 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.045174    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 00:35:03.045279 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.045207    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 00:35:03.045279 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.045247    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:35:03.045279 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.045275    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 00:35:03.045594 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.045313    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 00:35:03.046656 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.046634    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"]
Oct 31 00:35:03.070663 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.070642    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.070792 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.070675    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.070792 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.070699    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.070792 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.070724    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzskx\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-kube-api-access-gzskx\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.070914 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.070808    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5b040-977a-47cc-95f7-c1ef0370e09a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.070914 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.070861    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.070914 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.070891    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.172282 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.172248    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5b040-977a-47cc-95f7-c1ef0370e09a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.172282 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.172286    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.172518 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.172310    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.172518 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.172374    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.172518 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.172423    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.172518 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.172459    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.172518 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.172481    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzskx\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-kube-api-access-gzskx\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.173243 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.173217    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.174770 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.174750    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.174855 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.174782    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5b040-977a-47cc-95f7-c1ef0370e09a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.174855 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.174839    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.174962 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.174940    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.232753 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.232728    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.233746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.233720    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzskx\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-kube-api-access-gzskx\") pod \"istiod-openshift-gateway-7876bfcb74-6mctd\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.257449 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.257423    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:03.672200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:03.672179    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"]
Oct 31 00:35:03.674651 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:03.674625    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f5b040_977a_47cc_95f7_c1ef0370e09a.slice/crio-1980df4576c4a8cdf959648f6d14d80f6597ab54915c4676f55d390d74248479 WatchSource:0}: Error finding container 1980df4576c4a8cdf959648f6d14d80f6597ab54915c4676f55d390d74248479: Status 404 returned error can't find the container with id 1980df4576c4a8cdf959648f6d14d80f6597ab54915c4676f55d390d74248479
Oct 31 00:35:04.654803 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:04.654770    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" event={"ID":"f1f5b040-977a-47cc-95f7-c1ef0370e09a","Type":"ContainerStarted","Data":"1980df4576c4a8cdf959648f6d14d80f6597ab54915c4676f55d390d74248479"}
Oct 31 00:35:07.344716 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:07.344680    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:35:07.345109 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:07.344757    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:35:07.669117 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:07.669035    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" event={"ID":"f1f5b040-977a-47cc-95f7-c1ef0370e09a","Type":"ContainerStarted","Data":"fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0"}
Oct 31 00:35:07.669117 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:07.669097    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:07.787438 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:07.787392    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" podStartSLOduration=2.120000994 podStartE2EDuration="5.787371837s" totalImagesPullingTime="3.667370843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:07.786451505 +0000 UTC m=+316.341338943" watchObservedRunningTime="2026-10-31 00:35:07.787371837 +0000 UTC m=+316.342259269"
Oct 31 00:35:08.674656 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:08.674624    4888 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-6mctd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:35:08.675055 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:08.674678    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" podUID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:35:11.674036 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:11.674009    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:35:12.793465 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.793432    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"]
Oct 31 00:35:12.796584 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.796565    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850225 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850188    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850225 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850225    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850415 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850292    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850471 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850419    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d60c10e-c079-4f3b-a08e-843d51c9d457-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850542 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850468    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6x6\" (UniqueName: \"kubernetes.io/projected/8d60c10e-c079-4f3b-a08e-843d51c9d457-kube-api-access-4h6x6\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850542 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850502    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850542 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850519    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850693 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850545    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.850693 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.850577    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.892916 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.892892    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"]
Oct 31 00:35:12.938419 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.938392    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-q9j8d\""
Oct 31 00:35:12.951750 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.951728    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.951839 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.951767    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d60c10e-c079-4f3b-a08e-843d51c9d457-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.951839 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.951787    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6x6\" (UniqueName: \"kubernetes.io/projected/8d60c10e-c079-4f3b-a08e-843d51c9d457-kube-api-access-4h6x6\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.951839 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.951824    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.951974 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.951847    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.951974 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.951867    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952048 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952019    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952110 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952094    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952208 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952179    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952322 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952223    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952322 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952236    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952322 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952284    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952521 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952455    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.952521 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.952515    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d60c10e-c079-4f3b-a08e-843d51c9d457-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.954475 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.954452    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:12.954692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:12.954672    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:13.012498 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:13.012465    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d60c10e-c079-4f3b-a08e-843d51c9d457-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:13.012618 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:13.012502    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6x6\" (UniqueName: \"kubernetes.io/projected/8d60c10e-c079-4f3b-a08e-843d51c9d457-kube-api-access-4h6x6\") pod \"openshift-ai-inference-openshift-default-568c495856-xc7wn\" (UID: \"8d60c10e-c079-4f3b-a08e-843d51c9d457\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:13.108604 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:13.108530    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:13.522628 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:13.522538    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"]
Oct 31 00:35:13.525661 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:13.525636    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d60c10e_c079_4f3b_a08e_843d51c9d457.slice/crio-4604f072c7fb4c8ccf27f5611f790d8b5ec97cfc1235afdb36fc388d2aabd281 WatchSource:0}: Error finding container 4604f072c7fb4c8ccf27f5611f790d8b5ec97cfc1235afdb36fc388d2aabd281: Status 404 returned error can't find the container with id 4604f072c7fb4c8ccf27f5611f790d8b5ec97cfc1235afdb36fc388d2aabd281
Oct 31 00:35:13.691894 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:13.691862    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" event={"ID":"8d60c10e-c079-4f3b-a08e-843d51c9d457","Type":"ContainerStarted","Data":"4604f072c7fb4c8ccf27f5611f790d8b5ec97cfc1235afdb36fc388d2aabd281"}
Oct 31 00:35:17.630902 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:17.630869    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:35:17.631203 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:17.630935    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:35:17.631203 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:17.630963    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:35:17.710076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:17.710046    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" event={"ID":"8d60c10e-c079-4f3b-a08e-843d51c9d457","Type":"ContainerStarted","Data":"c6c4320b742831ea521529f3716fa6fde9f77dcfd9306d838436abb99b4c8a9a"}
Oct 31 00:35:17.851453 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:17.851386    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" podStartSLOduration=1.748718288 podStartE2EDuration="5.851370496s" totalImagesPullingTime="4.102652208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:17.850473606 +0000 UTC m=+326.405361045" watchObservedRunningTime="2026-10-31 00:35:17.851370496 +0000 UTC m=+326.406257931"
Oct 31 00:35:18.108899 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:18.108868    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:18.113477 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:18.113454    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:18.552718 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:35:18.552682    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:35:18.714404 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:18.714374    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:18.715237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:18.715219    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:35:22.821940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.821909    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"]
Oct 31 00:35:22.825019 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.824998    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:22.922481 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.922459    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9lbws\""
Oct 31 00:35:22.922636 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.922536    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 00:35:22.922636 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.922547    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 00:35:22.923825 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.923806    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"]
Oct 31 00:35:22.924704 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.924686    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"]
Oct 31 00:35:22.927969 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.927952    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:22.931001 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.930981    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:22.931095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.931024    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:22.931095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.931055    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wks\" (UniqueName: \"kubernetes.io/projected/e88a73c3-8cab-427a-ac29-9ffb608901a8-kube-api-access-p4wks\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:22.979000 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.978980    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-6trvl\""
Oct 31 00:35:22.989224 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:22.989202    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"]
Oct 31 00:35:23.021395 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.021371    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"]
Oct 31 00:35:23.024551 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.024536    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.032198 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032178    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wks\" (UniqueName: \"kubernetes.io/projected/e88a73c3-8cab-427a-ac29-9ffb608901a8-kube-api-access-p4wks\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:23.032287 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032209    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.032287 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032236    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.032414 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032383    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghjf\" (UniqueName: \"kubernetes.io/projected/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-kube-api-access-cghjf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.032460 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032448    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:23.032497 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032489    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:23.032766 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032750    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:23.032798 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.032785    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:23.079521 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.079476    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-2rmnm\""
Oct 31 00:35:23.082232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.082213    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"]
Oct 31 00:35:23.092672 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.092644    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wks\" (UniqueName: \"kubernetes.io/projected/e88a73c3-8cab-427a-ac29-9ffb608901a8-kube-api-access-p4wks\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:23.122556 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.122535    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"]
Oct 31 00:35:23.125819 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.125805    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.132996 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.132975    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.133087 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.133010    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.133087 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.133035    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghjf\" (UniqueName: \"kubernetes.io/projected/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-kube-api-access-cghjf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.133189 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.133142    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.133244 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.133209    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dt9b\" (UniqueName: \"kubernetes.io/projected/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-kube-api-access-8dt9b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.133334 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.133309    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.133427 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.133377    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.133480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.133424    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.139620 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.139580    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:23.179679 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.179645    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-9grqs\""
Oct 31 00:35:23.183013 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.182985    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"]
Oct 31 00:35:23.194768 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.194739    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghjf\" (UniqueName: \"kubernetes.io/projected/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-kube-api-access-cghjf\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.234798 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.234764    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.234950 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.234803    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wthh\" (UniqueName: \"kubernetes.io/projected/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-kube-api-access-6wthh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.234950 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.234842    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dt9b\" (UniqueName: \"kubernetes.io/projected/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-kube-api-access-8dt9b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.234950 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.234927    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.235051 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.235021    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.235051 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.235044    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.235304 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.235287    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.235343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.235317    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.242721 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.242691    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:23.297784 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.297758    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dt9b\" (UniqueName: \"kubernetes.io/projected/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-kube-api-access-8dt9b\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.335615 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.335545    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.335615 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.335581    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.335817 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.335628    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wthh\" (UniqueName: \"kubernetes.io/projected/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-kube-api-access-6wthh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.335994 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.335971    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.336063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.336032    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.339450 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.339433    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:23.396973 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.396944    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wthh\" (UniqueName: \"kubernetes.io/projected/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-kube-api-access-6wthh\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.443670 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.443638    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:23.511332 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.511309    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"]
Oct 31 00:35:23.514444 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:23.514407    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88a73c3_8cab_427a_ac29_9ffb608901a8.slice/crio-dca82be9d7f239bffd1aedcab8d97388a15c4e23313b05af9b503f77fdc624c4 WatchSource:0}: Error finding container dca82be9d7f239bffd1aedcab8d97388a15c4e23313b05af9b503f77fdc624c4: Status 404 returned error can't find the container with id dca82be9d7f239bffd1aedcab8d97388a15c4e23313b05af9b503f77fdc624c4
Oct 31 00:35:23.613294 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.613271    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"]
Oct 31 00:35:23.656298 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:23.656272    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8691e1c_5fbf_4bf4_88b5_ab58364a0651.slice/crio-fe5d837d3c0e5b672b363b8f28bf7aa3950237dd120981f57ac533819cebdd64 WatchSource:0}: Error finding container fe5d837d3c0e5b672b363b8f28bf7aa3950237dd120981f57ac533819cebdd64: Status 404 returned error can't find the container with id fe5d837d3c0e5b672b363b8f28bf7aa3950237dd120981f57ac533819cebdd64
Oct 31 00:35:23.706031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.706007    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"]
Oct 31 00:35:23.706865 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:23.706845    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f7e2e5_a69f_4147_8756_7aaa5e92658b.slice/crio-71be237504b7efdef3f8689464ba056d42075e0e8609733fc8e411b252c45848 WatchSource:0}: Error finding container 71be237504b7efdef3f8689464ba056d42075e0e8609733fc8e411b252c45848: Status 404 returned error can't find the container with id 71be237504b7efdef3f8689464ba056d42075e0e8609733fc8e411b252c45848
Oct 31 00:35:23.734319 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.734287    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv" event={"ID":"a8691e1c-5fbf-4bf4-88b5-ab58364a0651","Type":"ContainerStarted","Data":"2697f3fee546c724abc639a4913f1eb724de55673feecdb3b93d6b75155bcc26"}
Oct 31 00:35:23.734459 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.734342    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv" event={"ID":"a8691e1c-5fbf-4bf4-88b5-ab58364a0651","Type":"ContainerStarted","Data":"fe5d837d3c0e5b672b363b8f28bf7aa3950237dd120981f57ac533819cebdd64"}
Oct 31 00:35:23.735859 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.735834    4888 generic.go:356] "Generic (PLEG): container finished" podID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerID="7a75a04044bd90ba9e57c4a708f762c4a795242e648932157d4e97cb54d7647b" exitCode=0
Oct 31 00:35:23.735948 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.735869    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt" event={"ID":"e88a73c3-8cab-427a-ac29-9ffb608901a8","Type":"ContainerDied","Data":"7a75a04044bd90ba9e57c4a708f762c4a795242e648932157d4e97cb54d7647b"}
Oct 31 00:35:23.735948 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.735913    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt" event={"ID":"e88a73c3-8cab-427a-ac29-9ffb608901a8","Type":"ContainerStarted","Data":"dca82be9d7f239bffd1aedcab8d97388a15c4e23313b05af9b503f77fdc624c4"}
Oct 31 00:35:23.737441 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.737403    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc" event={"ID":"a7f7e2e5-a69f-4147-8756-7aaa5e92658b","Type":"ContainerStarted","Data":"71be237504b7efdef3f8689464ba056d42075e0e8609733fc8e411b252c45848"}
Oct 31 00:35:23.811209 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:23.811185    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"]
Oct 31 00:35:23.846087 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:23.846062    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e02a3a_165a_4887_a4e2_68ca2f148e1f.slice/crio-2dacedfe96bd983829d7da7fd3bc268e2b50674f4e6d9102692dfef6a7c8eeab WatchSource:0}: Error finding container 2dacedfe96bd983829d7da7fd3bc268e2b50674f4e6d9102692dfef6a7c8eeab: Status 404 returned error can't find the container with id 2dacedfe96bd983829d7da7fd3bc268e2b50674f4e6d9102692dfef6a7c8eeab
Oct 31 00:35:24.743655 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:24.743623    4888 generic.go:356] "Generic (PLEG): container finished" podID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerID="2697f3fee546c724abc639a4913f1eb724de55673feecdb3b93d6b75155bcc26" exitCode=0
Oct 31 00:35:24.743848 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:24.743685    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv" event={"ID":"a8691e1c-5fbf-4bf4-88b5-ab58364a0651","Type":"ContainerDied","Data":"2697f3fee546c724abc639a4913f1eb724de55673feecdb3b93d6b75155bcc26"}
Oct 31 00:35:24.745239 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:24.745209    4888 generic.go:356] "Generic (PLEG): container finished" podID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerID="e907cc833e3dd65390dc71b9870dc208340c8c596d7eb4d9864976980db3e359" exitCode=0
Oct 31 00:35:24.745375 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:24.745298    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr" event={"ID":"c9e02a3a-165a-4887-a4e2-68ca2f148e1f","Type":"ContainerDied","Data":"e907cc833e3dd65390dc71b9870dc208340c8c596d7eb4d9864976980db3e359"}
Oct 31 00:35:24.745375 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:24.745341    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr" event={"ID":"c9e02a3a-165a-4887-a4e2-68ca2f148e1f","Type":"ContainerStarted","Data":"2dacedfe96bd983829d7da7fd3bc268e2b50674f4e6d9102692dfef6a7c8eeab"}
Oct 31 00:35:24.747018 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:24.746969    4888 generic.go:356] "Generic (PLEG): container finished" podID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerID="337e5cc0d23d0ee79a0efdb29d16424115ef55f4cdd17b594ef290b8b1bb4385" exitCode=0
Oct 31 00:35:24.747073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:24.747022    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc" event={"ID":"a7f7e2e5-a69f-4147-8756-7aaa5e92658b","Type":"ContainerDied","Data":"337e5cc0d23d0ee79a0efdb29d16424115ef55f4cdd17b594ef290b8b1bb4385"}
Oct 31 00:35:25.752977 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:25.752948    4888 generic.go:356] "Generic (PLEG): container finished" podID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerID="cb0586578f4a0f4acf1f00714a30dbefc3fcce9eaa3ad5fe96a14c84a4e4401a" exitCode=0
Oct 31 00:35:25.753396 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:25.753040    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt" event={"ID":"e88a73c3-8cab-427a-ac29-9ffb608901a8","Type":"ContainerDied","Data":"cb0586578f4a0f4acf1f00714a30dbefc3fcce9eaa3ad5fe96a14c84a4e4401a"}
Oct 31 00:35:26.758558 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.758524    4888 generic.go:356] "Generic (PLEG): container finished" podID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerID="90460b778b44f498ec221f92c445b81a0a9914326cc6dcae6617575f6c6d1e46" exitCode=0
Oct 31 00:35:26.759003 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.758610    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv" event={"ID":"a8691e1c-5fbf-4bf4-88b5-ab58364a0651","Type":"ContainerDied","Data":"90460b778b44f498ec221f92c445b81a0a9914326cc6dcae6617575f6c6d1e46"}
Oct 31 00:35:26.760565 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.760542    4888 generic.go:356] "Generic (PLEG): container finished" podID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerID="fadd6113dc1550c253e05bc112a86f46470ea4e1a8b0a512cefa114c82b29ad1" exitCode=0
Oct 31 00:35:26.760677 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.760620    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr" event={"ID":"c9e02a3a-165a-4887-a4e2-68ca2f148e1f","Type":"ContainerDied","Data":"fadd6113dc1550c253e05bc112a86f46470ea4e1a8b0a512cefa114c82b29ad1"}
Oct 31 00:35:26.763112 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.763084    4888 generic.go:356] "Generic (PLEG): container finished" podID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerID="47cea7063a35b692d5f728588853e2c8a36be1ec56dffcb8cb3a9e5984c1fe26" exitCode=0
Oct 31 00:35:26.763210 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.763185    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt" event={"ID":"e88a73c3-8cab-427a-ac29-9ffb608901a8","Type":"ContainerDied","Data":"47cea7063a35b692d5f728588853e2c8a36be1ec56dffcb8cb3a9e5984c1fe26"}
Oct 31 00:35:26.764952 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.764933    4888 generic.go:356] "Generic (PLEG): container finished" podID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerID="476f3d9c5f40a2c305e3b7ce9c03c03d7f25d77b162aa7d373f0fea2372b0645" exitCode=0
Oct 31 00:35:26.765023 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:26.764991    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc" event={"ID":"a7f7e2e5-a69f-4147-8756-7aaa5e92658b","Type":"ContainerDied","Data":"476f3d9c5f40a2c305e3b7ce9c03c03d7f25d77b162aa7d373f0fea2372b0645"}
Oct 31 00:35:27.771276 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.771247    4888 generic.go:356] "Generic (PLEG): container finished" podID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerID="301bc56288f5d11dd1183829aab6dfff5775cb320da00f4573c7c929ca577470" exitCode=0
Oct 31 00:35:27.771725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.771344    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc" event={"ID":"a7f7e2e5-a69f-4147-8756-7aaa5e92658b","Type":"ContainerDied","Data":"301bc56288f5d11dd1183829aab6dfff5775cb320da00f4573c7c929ca577470"}
Oct 31 00:35:27.773305 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.773283    4888 generic.go:356] "Generic (PLEG): container finished" podID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerID="203c97d7caba260a5fdcc23253a45cbf0203d3fcd3f50bf8c0228b0e541b6e5c" exitCode=0
Oct 31 00:35:27.773439 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.773381    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv" event={"ID":"a8691e1c-5fbf-4bf4-88b5-ab58364a0651","Type":"ContainerDied","Data":"203c97d7caba260a5fdcc23253a45cbf0203d3fcd3f50bf8c0228b0e541b6e5c"}
Oct 31 00:35:27.775278 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.775260    4888 generic.go:356] "Generic (PLEG): container finished" podID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerID="d7a4784b5657a6cc08c8781c77a22b5ff448cfe0038ee6930d8c4893326c6e99" exitCode=0
Oct 31 00:35:27.775333 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.775290    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr" event={"ID":"c9e02a3a-165a-4887-a4e2-68ca2f148e1f","Type":"ContainerDied","Data":"d7a4784b5657a6cc08c8781c77a22b5ff448cfe0038ee6930d8c4893326c6e99"}
Oct 31 00:35:27.891411 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.891388    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:27.970837 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.970807    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-util\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-util\") pod \"e88a73c3-8cab-427a-ac29-9ffb608901a8\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") "
Oct 31 00:35:27.970995 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.970851    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-bundle\") pod \"e88a73c3-8cab-427a-ac29-9ffb608901a8\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") "
Oct 31 00:35:27.970995 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.970885    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88a73c3-8cab-427a-ac29-9ffb608901a8-kube-api-access-p4wks\" (UniqueName: \"kubernetes.io/projected/e88a73c3-8cab-427a-ac29-9ffb608901a8-kube-api-access-p4wks\") pod \"e88a73c3-8cab-427a-ac29-9ffb608901a8\" (UID: \"e88a73c3-8cab-427a-ac29-9ffb608901a8\") "
Oct 31 00:35:27.971586 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.971561    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-bundle" pod "e88a73c3-8cab-427a-ac29-9ffb608901a8" (UID: "e88a73c3-8cab-427a-ac29-9ffb608901a8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:27.972990 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.972970    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88a73c3-8cab-427a-ac29-9ffb608901a8-kube-api-access-p4wks" pod "e88a73c3-8cab-427a-ac29-9ffb608901a8" (UID: "e88a73c3-8cab-427a-ac29-9ffb608901a8"). InnerVolumeSpecName "kube-api-access-p4wks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:27.975922 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:27.975883    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-util" pod "e88a73c3-8cab-427a-ac29-9ffb608901a8" (UID: "e88a73c3-8cab-427a-ac29-9ffb608901a8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:28.072453 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.072377    4888 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-bundle\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:28.072453 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.072405    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4wks\" (UniqueName: \"kubernetes.io/projected/e88a73c3-8cab-427a-ac29-9ffb608901a8-kube-api-access-p4wks\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:28.072453 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.072419    4888 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e88a73c3-8cab-427a-ac29-9ffb608901a8-util\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:28.781020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.780984    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt" event={"ID":"e88a73c3-8cab-427a-ac29-9ffb608901a8","Type":"ContainerDied","Data":"dca82be9d7f239bffd1aedcab8d97388a15c4e23313b05af9b503f77fdc624c4"}
Oct 31 00:35:28.781436 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.781036    4888 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dca82be9d7f239bffd1aedcab8d97388a15c4e23313b05af9b503f77fdc624c4"
Oct 31 00:35:28.781436 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.780997    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah48vt"
Oct 31 00:35:28.914217 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.914193    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:28.953283 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.953263    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:28.955930 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:28.955913    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:29.081804 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.081730    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-util\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-util\") pod \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") "
Oct 31 00:35:29.081804 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.081795    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-bundle\") pod \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") "
Oct 31 00:35:29.082188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.081828    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-bundle\") pod \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") "
Oct 31 00:35:29.082188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.081882    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-kube-api-access-8dt9b\" (UniqueName: \"kubernetes.io/projected/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-kube-api-access-8dt9b\") pod \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") "
Oct 31 00:35:29.082188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.081911    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-util\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-util\") pod \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") "
Oct 31 00:35:29.082188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.081937    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-kube-api-access-6wthh\" (UniqueName: \"kubernetes.io/projected/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-kube-api-access-6wthh\") pod \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\" (UID: \"c9e02a3a-165a-4887-a4e2-68ca2f148e1f\") "
Oct 31 00:35:29.082188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.081992    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-bundle\") pod \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") "
Oct 31 00:35:29.082188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.082025    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-kube-api-access-cghjf\" (UniqueName: \"kubernetes.io/projected/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-kube-api-access-cghjf\") pod \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\" (UID: \"a8691e1c-5fbf-4bf4-88b5-ab58364a0651\") "
Oct 31 00:35:29.082188 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.082049    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-util\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-util\") pod \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\" (UID: \"a7f7e2e5-a69f-4147-8756-7aaa5e92658b\") "
Oct 31 00:35:29.082569 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.082420    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-bundle" pod "a7f7e2e5-a69f-4147-8756-7aaa5e92658b" (UID: "a7f7e2e5-a69f-4147-8756-7aaa5e92658b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:29.082646 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.082569    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-bundle" pod "c9e02a3a-165a-4887-a4e2-68ca2f148e1f" (UID: "c9e02a3a-165a-4887-a4e2-68ca2f148e1f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:29.082646 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.082589    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-bundle" pod "a8691e1c-5fbf-4bf4-88b5-ab58364a0651" (UID: "a8691e1c-5fbf-4bf4-88b5-ab58364a0651"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:29.084140 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.084117    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-kube-api-access-8dt9b" pod "a7f7e2e5-a69f-4147-8756-7aaa5e92658b" (UID: "a7f7e2e5-a69f-4147-8756-7aaa5e92658b"). InnerVolumeSpecName "kube-api-access-8dt9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:29.084255 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.084202    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-kube-api-access-6wthh" pod "c9e02a3a-165a-4887-a4e2-68ca2f148e1f" (UID: "c9e02a3a-165a-4887-a4e2-68ca2f148e1f"). InnerVolumeSpecName "kube-api-access-6wthh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:29.084473 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.084452    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-kube-api-access-cghjf" pod "a8691e1c-5fbf-4bf4-88b5-ab58364a0651" (UID: "a8691e1c-5fbf-4bf4-88b5-ab58364a0651"). InnerVolumeSpecName "kube-api-access-cghjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:29.183434 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.183407    4888 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-bundle\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.183434 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.183427    4888 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-bundle\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.183434 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.183438    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dt9b\" (UniqueName: \"kubernetes.io/projected/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-kube-api-access-8dt9b\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.183640 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.183448    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wthh\" (UniqueName: \"kubernetes.io/projected/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-kube-api-access-6wthh\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.183640 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.183456    4888 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-bundle\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.183640 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.183464    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghjf\" (UniqueName: \"kubernetes.io/projected/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-kube-api-access-cghjf\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.372330 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.372226    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-util" pod "c9e02a3a-165a-4887-a4e2-68ca2f148e1f" (UID: "c9e02a3a-165a-4887-a4e2-68ca2f148e1f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:29.372974 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.372940    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-util" pod "a8691e1c-5fbf-4bf4-88b5-ab58364a0651" (UID: "a8691e1c-5fbf-4bf4-88b5-ab58364a0651"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:29.373942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.373922    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-util" pod "a7f7e2e5-a69f-4147-8756-7aaa5e92658b" (UID: "a7f7e2e5-a69f-4147-8756-7aaa5e92658b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:29.384805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.384788    4888 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9e02a3a-165a-4887-a4e2-68ca2f148e1f-util\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.384856 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.384807    4888 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a8691e1c-5fbf-4bf4-88b5-ab58364a0651-util\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.384856 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.384815    4888 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a7f7e2e5-a69f-4147-8756-7aaa5e92658b-util\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:35:29.786544 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.786517    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc"
Oct 31 00:35:29.786544 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.786516    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xdjtc" event={"ID":"a7f7e2e5-a69f-4147-8756-7aaa5e92658b","Type":"ContainerDied","Data":"71be237504b7efdef3f8689464ba056d42075e0e8609733fc8e411b252c45848"}
Oct 31 00:35:29.787054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.786575    4888 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71be237504b7efdef3f8689464ba056d42075e0e8609733fc8e411b252c45848"
Oct 31 00:35:29.788523 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.788498    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv" event={"ID":"a8691e1c-5fbf-4bf4-88b5-ab58364a0651","Type":"ContainerDied","Data":"fe5d837d3c0e5b672b363b8f28bf7aa3950237dd120981f57ac533819cebdd64"}
Oct 31 00:35:29.788651 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.788541    4888 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5d837d3c0e5b672b363b8f28bf7aa3950237dd120981f57ac533819cebdd64"
Oct 31 00:35:29.788761 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.788652    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6phtmv"
Oct 31 00:35:29.790596 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.790527    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr" event={"ID":"c9e02a3a-165a-4887-a4e2-68ca2f148e1f","Type":"ContainerDied","Data":"2dacedfe96bd983829d7da7fd3bc268e2b50674f4e6d9102692dfef6a7c8eeab"}
Oct 31 00:35:29.790596 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.790572    4888 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dacedfe96bd983829d7da7fd3bc268e2b50674f4e6d9102692dfef6a7c8eeab"
Oct 31 00:35:29.790596 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:29.790590    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4crqrsr"
Oct 31 00:35:41.365466 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365426    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-66498fdf6c-m4lqp"]
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365831    4888 cpu_manager.go:414] "containerMap: removing container" podUID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerName="extract"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365848    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerName="extract"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365865    4888 cpu_manager.go:414] "containerMap: removing container" podUID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerName="extract"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365871    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerName="extract"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365882    4888 cpu_manager.go:414] "containerMap: removing container" podUID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365888    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365911    4888 cpu_manager.go:414] "containerMap: removing container" podUID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365918    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365929    4888 cpu_manager.go:414] "containerMap: removing container" podUID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365935    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365945    4888 cpu_manager.go:414] "containerMap: removing container" podUID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerName="extract"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365952    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerName="extract"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365961    4888 cpu_manager.go:414] "containerMap: removing container" podUID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365967    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365982    4888 cpu_manager.go:414] "containerMap: removing container" podUID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.365989    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366005    4888 cpu_manager.go:414] "containerMap: removing container" podUID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366011    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerName="util"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366020    4888 cpu_manager.go:414] "containerMap: removing container" podUID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366026    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366034    4888 cpu_manager.go:414] "containerMap: removing container" podUID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366041    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerName="pull"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366054    4888 cpu_manager.go:414] "containerMap: removing container" podUID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerName="extract"
Oct 31 00:35:41.366076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366060    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerName="extract"
Oct 31 00:35:41.367190 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366164    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f7e2e5-a69f-4147-8756-7aaa5e92658b" containerName="extract"
Oct 31 00:35:41.367190 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366178    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88a73c3-8cab-427a-ac29-9ffb608901a8" containerName="extract"
Oct 31 00:35:41.367190 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366189    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e02a3a-165a-4887-a4e2-68ca2f148e1f" containerName="extract"
Oct 31 00:35:41.367190 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.366201    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8691e1c-5fbf-4bf4-88b5-ab58364a0651" containerName="extract"
Oct 31 00:35:41.373077 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.373055    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp"
Oct 31 00:35:41.471048 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.471023    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-qvgn7\""
Oct 31 00:35:41.471210 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.471081    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 00:35:41.471210 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.471110    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 00:35:41.472651 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.472627    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-66498fdf6c-m4lqp"]
Oct 31 00:35:41.473045 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.473018    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgv7f\" (UniqueName: \"kubernetes.io/projected/c0a4397b-047b-40b6-8b72-acaca7e6d2af-kube-api-access-xgv7f\") pod \"authorino-operator-66498fdf6c-m4lqp\" (UID: \"c0a4397b-047b-40b6-8b72-acaca7e6d2af\") " pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp"
Oct 31 00:35:41.573929 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.573900    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgv7f\" (UniqueName: \"kubernetes.io/projected/c0a4397b-047b-40b6-8b72-acaca7e6d2af-kube-api-access-xgv7f\") pod \"authorino-operator-66498fdf6c-m4lqp\" (UID: \"c0a4397b-047b-40b6-8b72-acaca7e6d2af\") " pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp"
Oct 31 00:35:41.634146 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.634072    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgv7f\" (UniqueName: \"kubernetes.io/projected/c0a4397b-047b-40b6-8b72-acaca7e6d2af-kube-api-access-xgv7f\") pod \"authorino-operator-66498fdf6c-m4lqp\" (UID: \"c0a4397b-047b-40b6-8b72-acaca7e6d2af\") " pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp"
Oct 31 00:35:41.684730 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:41.684704    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp"
Oct 31 00:35:42.063180 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:42.063157    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-66498fdf6c-m4lqp"]
Oct 31 00:35:42.065083 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:42.065050    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a4397b_047b_40b6_8b72_acaca7e6d2af.slice/crio-4ae298549bba6f19f8fe63576ccc80cf346ec3b903507b795d78c5114630fb07 WatchSource:0}: Error finding container 4ae298549bba6f19f8fe63576ccc80cf346ec3b903507b795d78c5114630fb07: Status 404 returned error can't find the container with id 4ae298549bba6f19f8fe63576ccc80cf346ec3b903507b795d78c5114630fb07
Oct 31 00:35:42.067206 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:42.067189    4888 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:35:42.848617 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:42.848583    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp" event={"ID":"c0a4397b-047b-40b6-8b72-acaca7e6d2af","Type":"ContainerStarted","Data":"4ae298549bba6f19f8fe63576ccc80cf346ec3b903507b795d78c5114630fb07"}
Oct 31 00:35:46.869002 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:46.868961    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp" event={"ID":"c0a4397b-047b-40b6-8b72-acaca7e6d2af","Type":"ContainerStarted","Data":"3eac8ea3882c016cd7f73a238f3eef6399a33d60368af5d9a8ee431f57aed4a2"}
Oct 31 00:35:46.869505 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:46.869054    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp"
Oct 31 00:35:46.995966 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:46.995915    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp" podStartSLOduration=1.738857299 podStartE2EDuration="5.995901282s" totalImagesPullingTime="4.257043983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:46.994028156 +0000 UTC m=+355.548915593" watchObservedRunningTime="2026-10-31 00:35:46.995901282 +0000 UTC m=+355.550788720"
Oct 31 00:35:47.748335 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:47.748301    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"]
Oct 31 00:35:47.750400 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:47.750384    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"
Oct 31 00:35:47.854627 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:47.854601    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cwbwg\""
Oct 31 00:35:47.856250 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:47.856232    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"]
Oct 31 00:35:47.924872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:47.924837    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhkd\" (UniqueName: \"kubernetes.io/projected/fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa-kube-api-access-kxhkd\") pod \"limitador-operator-controller-manager-69b7bb8c5c-wpvmf\" (UID: \"fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa\") " pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"
Oct 31 00:35:48.026140 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:48.026053    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhkd\" (UniqueName: \"kubernetes.io/projected/fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa-kube-api-access-kxhkd\") pod \"limitador-operator-controller-manager-69b7bb8c5c-wpvmf\" (UID: \"fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa\") " pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"
Oct 31 00:35:48.086480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:48.086451    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhkd\" (UniqueName: \"kubernetes.io/projected/fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa-kube-api-access-kxhkd\") pod \"limitador-operator-controller-manager-69b7bb8c5c-wpvmf\" (UID: \"fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa\") " pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"
Oct 31 00:35:48.361412 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:48.361318    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"
Oct 31 00:35:48.550674 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:35:48.550643    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:35:48.729828 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:48.729802    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"]
Oct 31 00:35:48.732803 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:35:48.732773    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5fdab6_2b4e_41f9_9d83_25cd01acd3fa.slice/crio-64d3dbe107cdbaf2c6b35440ea48561549de02c1ff62ac02c6761f8546076d07 WatchSource:0}: Error finding container 64d3dbe107cdbaf2c6b35440ea48561549de02c1ff62ac02c6761f8546076d07: Status 404 returned error can't find the container with id 64d3dbe107cdbaf2c6b35440ea48561549de02c1ff62ac02c6761f8546076d07
Oct 31 00:35:48.879221 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:48.879186    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf" event={"ID":"fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa","Type":"ContainerStarted","Data":"64d3dbe107cdbaf2c6b35440ea48561549de02c1ff62ac02c6761f8546076d07"}
Oct 31 00:35:52.899992 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:52.899953    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf" event={"ID":"fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa","Type":"ContainerStarted","Data":"d1a45787b41ec3f5b813ce160e44292c2a304154e31e45bebcead7d8e419ad99"}
Oct 31 00:35:52.900415 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:52.900159    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"
Oct 31 00:35:53.021085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:53.020220    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf" podStartSLOduration=2.7555331069999998 podStartE2EDuration="6.020201121s" totalImagesPullingTime="3.264668014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:53.01996239 +0000 UTC m=+361.574849828" watchObservedRunningTime="2026-10-31 00:35:53.020201121 +0000 UTC m=+361.575088561"
Oct 31 00:35:57.875827 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:35:57.875799    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-66498fdf6c-m4lqp"
Oct 31 00:36:03.907930 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:03.907897    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-69b7bb8c5c-wpvmf"
Oct 31 00:36:18.551179 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:36:18.551143    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:36:45.686942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:45.686914    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pttdq"]
Oct 31 00:36:45.690998 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:45.690975    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq"
Oct 31 00:36:45.742784 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:45.742758    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pttdq"]
Oct 31 00:36:45.758628 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:45.758602    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfcrg\" (UniqueName: \"kubernetes.io/projected/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b-kube-api-access-xfcrg\") pod \"authorino-56c6c9ddd5-pttdq\" (UID: \"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b\") " pod="kuadrant-system/authorino-56c6c9ddd5-pttdq"
Oct 31 00:36:45.787891 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:45.787873    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9xw28\""
Oct 31 00:36:45.858953 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:45.858922    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfcrg\" (UniqueName: \"kubernetes.io/projected/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b-kube-api-access-xfcrg\") pod \"authorino-56c6c9ddd5-pttdq\" (UID: \"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b\") " pod="kuadrant-system/authorino-56c6c9ddd5-pttdq"
Oct 31 00:36:45.918932 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:45.918902    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfcrg\" (UniqueName: \"kubernetes.io/projected/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b-kube-api-access-xfcrg\") pod \"authorino-56c6c9ddd5-pttdq\" (UID: \"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b\") " pod="kuadrant-system/authorino-56c6c9ddd5-pttdq"
Oct 31 00:36:46.002698 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:46.002672    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq"
Oct 31 00:36:46.365545 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:46.365414    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pttdq"]
Oct 31 00:36:46.367679 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:36:46.367654    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5678bbe1_1bc3_4f7f_8c73_ea3a88c2041b.slice/crio-c6763480a3f87f047676e49fd8b7e5eba56d3e936d48db54864ec96f800d9790 WatchSource:0}: Error finding container c6763480a3f87f047676e49fd8b7e5eba56d3e936d48db54864ec96f800d9790: Status 404 returned error can't find the container with id c6763480a3f87f047676e49fd8b7e5eba56d3e936d48db54864ec96f800d9790
Oct 31 00:36:47.140618 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:47.140578    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq" event={"ID":"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b","Type":"ContainerStarted","Data":"c6763480a3f87f047676e49fd8b7e5eba56d3e936d48db54864ec96f800d9790"}
Oct 31 00:36:48.552718 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:36:48.552687    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:36:49.152016 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:49.151979    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq" event={"ID":"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b","Type":"ContainerStarted","Data":"bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957"}
Oct 31 00:36:49.264166 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:36:49.264066    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq" podStartSLOduration=1.629792463 podStartE2EDuration="4.264052307s" totalImagesPullingTime="2.634259844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:49.262658583 +0000 UTC m=+417.817546023" watchObservedRunningTime="2026-10-31 00:36:49.264052307 +0000 UTC m=+417.818939773"
Oct 31 00:37:13.917933 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:13.917897    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557c8f8584-vvnzr"]
Oct 31 00:37:13.921057 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:13.921041    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:13.972513 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:13.972487    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 00:37:14.020235 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.020205    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557c8f8584-vvnzr"]
Oct 31 00:37:14.072257 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.072227    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb687462-eacd-481c-a7a6-bf64ea2e3d29-tls-cert\") pod \"authorino-557c8f8584-vvnzr\" (UID: \"eb687462-eacd-481c-a7a6-bf64ea2e3d29\") " pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:14.072257 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.072261    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8qlc\" (UniqueName: \"kubernetes.io/projected/eb687462-eacd-481c-a7a6-bf64ea2e3d29-kube-api-access-m8qlc\") pod \"authorino-557c8f8584-vvnzr\" (UID: \"eb687462-eacd-481c-a7a6-bf64ea2e3d29\") " pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:14.173005 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.172921    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb687462-eacd-481c-a7a6-bf64ea2e3d29-tls-cert\") pod \"authorino-557c8f8584-vvnzr\" (UID: \"eb687462-eacd-481c-a7a6-bf64ea2e3d29\") " pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:14.173005 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.172958    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8qlc\" (UniqueName: \"kubernetes.io/projected/eb687462-eacd-481c-a7a6-bf64ea2e3d29-kube-api-access-m8qlc\") pod \"authorino-557c8f8584-vvnzr\" (UID: \"eb687462-eacd-481c-a7a6-bf64ea2e3d29\") " pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:14.175492 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.175468    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/eb687462-eacd-481c-a7a6-bf64ea2e3d29-tls-cert\") pod \"authorino-557c8f8584-vvnzr\" (UID: \"eb687462-eacd-481c-a7a6-bf64ea2e3d29\") " pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:14.233731 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.233704    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8qlc\" (UniqueName: \"kubernetes.io/projected/eb687462-eacd-481c-a7a6-bf64ea2e3d29-kube-api-access-m8qlc\") pod \"authorino-557c8f8584-vvnzr\" (UID: \"eb687462-eacd-481c-a7a6-bf64ea2e3d29\") " pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:14.532149 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.532104    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557c8f8584-vvnzr"
Oct 31 00:37:14.898359 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:14.898261    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557c8f8584-vvnzr"]
Oct 31 00:37:14.901095 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:37:14.901068    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb687462_eacd_481c_a7a6_bf64ea2e3d29.slice/crio-30852f46a9f1e79d99d8bfeb0175b1520ba37975fda3e9c4fe99abc3ad094861 WatchSource:0}: Error finding container 30852f46a9f1e79d99d8bfeb0175b1520ba37975fda3e9c4fe99abc3ad094861: Status 404 returned error can't find the container with id 30852f46a9f1e79d99d8bfeb0175b1520ba37975fda3e9c4fe99abc3ad094861
Oct 31 00:37:15.267324 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:15.267288    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557c8f8584-vvnzr" event={"ID":"eb687462-eacd-481c-a7a6-bf64ea2e3d29","Type":"ContainerStarted","Data":"30852f46a9f1e79d99d8bfeb0175b1520ba37975fda3e9c4fe99abc3ad094861"}
Oct 31 00:37:17.277742 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.277708    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557c8f8584-vvnzr" event={"ID":"eb687462-eacd-481c-a7a6-bf64ea2e3d29","Type":"ContainerStarted","Data":"3844c062a2284ebf107c40e0f9bb1666473c8d25c9e1b1527d295cdad62e5938"}
Oct 31 00:37:17.389467 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.389418    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-557c8f8584-vvnzr" podStartSLOduration=2.86769607 podStartE2EDuration="4.389404597s" totalImagesPullingTime="1.521708527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:17.389053496 +0000 UTC m=+445.943940933" watchObservedRunningTime="2026-10-31 00:37:17.389404597 +0000 UTC m=+445.944292036"
Oct 31 00:37:17.418591 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.418562    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pttdq"]
Oct 31 00:37:17.418768 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.418738    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq" podUID="5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b" containerName="authorino" containerID="cri-o://bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957" gracePeriod=30
Oct 31 00:37:17.669114 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.669088    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq"
Oct 31 00:37:17.804800 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.804725    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b-kube-api-access-xfcrg\" (UniqueName: \"kubernetes.io/projected/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b-kube-api-access-xfcrg\") pod \"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b\" (UID: \"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b\") "
Oct 31 00:37:17.807005 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.806974    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b-kube-api-access-xfcrg" pod "5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b" (UID: "5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b"). InnerVolumeSpecName "kube-api-access-xfcrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:17.905921 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:17.905892    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfcrg\" (UniqueName: \"kubernetes.io/projected/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b-kube-api-access-xfcrg\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:18.282987 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.282952    4888 generic.go:356] "Generic (PLEG): container finished" podID="5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b" containerID="bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957" exitCode=0
Oct 31 00:37:18.283392 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.282999    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq"
Oct 31 00:37:18.283392 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.283040    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq" event={"ID":"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b","Type":"ContainerDied","Data":"bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957"}
Oct 31 00:37:18.283392 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.283082    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-pttdq" event={"ID":"5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b","Type":"ContainerDied","Data":"c6763480a3f87f047676e49fd8b7e5eba56d3e936d48db54864ec96f800d9790"}
Oct 31 00:37:18.283392 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.283102    4888 scope.go:122] "RemoveContainer" containerID="bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957"
Oct 31 00:37:18.291472 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.291455    4888 scope.go:122] "RemoveContainer" containerID="bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957"
Oct 31 00:37:18.291723 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:37:18.291701    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957\": container with ID starting with bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957 not found: ID does not exist" containerID="bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957"
Oct 31 00:37:18.291777 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.291734    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957"} err="failed to get container status \"bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957\": rpc error: code = NotFound desc = could not find container \"bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957\": container with ID starting with bcf47a1720ed65c09e6ba7ccbefefc25a033ae10b8984619d065bd4b151a9957 not found: ID does not exist"
Oct 31 00:37:18.441665 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.441633    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pttdq"]
Oct 31 00:37:18.447512 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:18.447490    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-pttdq"]
Oct 31 00:37:18.550683 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:37:18.550606    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:37:19.552928 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:19.552901    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b" path="/var/lib/kubelet/pods/5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b/volumes"
Oct 31 00:37:36.137638 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.137611    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"]
Oct 31 00:37:36.138031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.137912    4888 cpu_manager.go:414] "containerMap: removing container" podUID="5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b" containerName="authorino"
Oct 31 00:37:36.138031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.137924    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b" containerName="authorino"
Oct 31 00:37:36.138031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.137989    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="5678bbe1-1bc3-4f7f-8c73-ea3a88c2041b" containerName="authorino"
Oct 31 00:37:36.141920 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.141903    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.243034 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.243007    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.243034 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.243040    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhkfj\" (UniqueName: \"kubernetes.io/projected/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-kube-api-access-xhkfj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.243216 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.243066    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.243216 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.243166    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.243216 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.243190    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.243216 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.243205    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.243333 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.243229    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.260417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.260390    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"]
Oct 31 00:37:36.343724 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.343686    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.343910 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.343739    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.343910 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.343757    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.343910 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.343771    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.343910 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.343787    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.343910 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.343834    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.343910 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.343854    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhkfj\" (UniqueName: \"kubernetes.io/projected/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-kube-api-access-xhkfj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.344540 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.344512    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.346307 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.346279    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.346449 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.346430    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.346511 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.346467    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.346511 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.346497    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.403668 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.403604    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhkfj\" (UniqueName: \"kubernetes.io/projected/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-kube-api-access-xhkfj\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.403822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.403805    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b60237-5b3f-4d48-a1df-b1fa8a211dd4-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-47szr\" (UID: \"81b60237-5b3f-4d48-a1df-b1fa8a211dd4\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.453606 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.453583    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:36.871196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.871174    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"]
Oct 31 00:37:36.873392 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:37:36.873367    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b60237_5b3f_4d48_a1df_b1fa8a211dd4.slice/crio-9e4b7050ca40e69145c596d7366c1061241e331a0abdb5830a9ecb338b273b38 WatchSource:0}: Error finding container 9e4b7050ca40e69145c596d7366c1061241e331a0abdb5830a9ecb338b273b38: Status 404 returned error can't find the container with id 9e4b7050ca40e69145c596d7366c1061241e331a0abdb5830a9ecb338b273b38
Oct 31 00:37:36.875324 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.875295    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:37:36.875428 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:36.875373    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:37:37.370177 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:37.370142    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr" event={"ID":"81b60237-5b3f-4d48-a1df-b1fa8a211dd4","Type":"ContainerStarted","Data":"635e8093b054fe3220760b4b572a152e6f0dea818d6b9fec5b4e68f3e823b424"}
Oct 31 00:37:37.370665 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:37.370191    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr" event={"ID":"81b60237-5b3f-4d48-a1df-b1fa8a211dd4","Type":"ContainerStarted","Data":"9e4b7050ca40e69145c596d7366c1061241e331a0abdb5830a9ecb338b273b38"}
Oct 31 00:37:37.370665 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:37.370208    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:37.371970 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:37.371946    4888 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-47szr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:37:37.372092 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:37.371997    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr" podUID="81b60237-5b3f-4d48-a1df-b1fa8a211dd4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:37:37.511727 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:37.511674    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr" podStartSLOduration=1.511661563 podStartE2EDuration="1.511661563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:37.502622424 +0000 UTC m=+466.057509875" watchObservedRunningTime="2026-10-31 00:37:37.511661563 +0000 UTC m=+466.066549001"
Oct 31 00:37:38.375572 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:38.375544    4888 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-47szr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:37:38.375934 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:38.375588    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr" podUID="81b60237-5b3f-4d48-a1df-b1fa8a211dd4" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:37:41.376065 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.376037    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-47szr"
Oct 31 00:37:41.591105 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.591074    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"]
Oct 31 00:37:41.591343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.591320    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" podUID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" containerName="discovery" containerID="cri-o://fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0" gracePeriod=30
Oct 31 00:37:41.673142 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.673119    4888 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-6mctd container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.20:8080/ready\": dial tcp 10.134.0.20:8080: connect: connection refused" start-of-body=
Oct 31 00:37:41.673244 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.673165    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" podUID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.20:8080/ready\": dial tcp 10.134.0.20:8080: connect: connection refused"
Oct 31 00:37:41.839878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.839856    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:37:41.883586 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.883563    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f5b040-977a-47cc-95f7-c1ef0370e09a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5b040-977a-47cc-95f7-c1ef0370e09a-local-certs\") pod \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") "
Oct 31 00:37:41.883680 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.883601    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-kubeconfig\") pod \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") "
Oct 31 00:37:41.883680 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.883623    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-kube-api-access-gzskx\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-kube-api-access-gzskx\") pod \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") "
Oct 31 00:37:41.883680 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.883654    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-token\") pod \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") "
Oct 31 00:37:41.883820 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.883718    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-dns-cert\") pod \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") "
Oct 31 00:37:41.883820 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.883751    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-cacerts\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-cacerts\") pod \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") "
Oct 31 00:37:41.883921 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.883901    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-ca-configmap\") pod \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\" (UID: \"f1f5b040-977a-47cc-95f7-c1ef0370e09a\") "
Oct 31 00:37:41.884786 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.884755    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-ca-configmap" pod "f1f5b040-977a-47cc-95f7-c1ef0370e09a" (UID: "f1f5b040-977a-47cc-95f7-c1ef0370e09a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:37:41.886090 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.886055    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-kubeconfig" pod "f1f5b040-977a-47cc-95f7-c1ef0370e09a" (UID: "f1f5b040-977a-47cc-95f7-c1ef0370e09a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:37:41.886200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.886136    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-cacerts" pod "f1f5b040-977a-47cc-95f7-c1ef0370e09a" (UID: "f1f5b040-977a-47cc-95f7-c1ef0370e09a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:37:41.886200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.886156    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-kube-api-access-gzskx" pod "f1f5b040-977a-47cc-95f7-c1ef0370e09a" (UID: "f1f5b040-977a-47cc-95f7-c1ef0370e09a"). InnerVolumeSpecName "kube-api-access-gzskx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:41.886313 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.886234    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-token" pod "f1f5b040-977a-47cc-95f7-c1ef0370e09a" (UID: "f1f5b040-977a-47cc-95f7-c1ef0370e09a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:41.886381 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.886319    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-dns-cert" pod "f1f5b040-977a-47cc-95f7-c1ef0370e09a" (UID: "f1f5b040-977a-47cc-95f7-c1ef0370e09a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:37:41.886424 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.886387    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f5b040-977a-47cc-95f7-c1ef0370e09a-local-certs" pod "f1f5b040-977a-47cc-95f7-c1ef0370e09a" (UID: "f1f5b040-977a-47cc-95f7-c1ef0370e09a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:37:41.984822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.984781    4888 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1f5b040-977a-47cc-95f7-c1ef0370e09a-local-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:41.984822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.984816    4888 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-kubeconfig\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:41.984822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.984825    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzskx\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-kube-api-access-gzskx\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:41.984822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.984834    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:41.985067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.984842    4888 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-dns-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:41.985067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.984850    4888 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f1f5b040-977a-47cc-95f7-c1ef0370e09a-cacerts\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:41.985067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:41.984858    4888 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f1f5b040-977a-47cc-95f7-c1ef0370e09a-istio-csr-ca-configmap\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:37:42.393710 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.393621    4888 generic.go:356] "Generic (PLEG): container finished" podID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" containerID="fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0" exitCode=0
Oct 31 00:37:42.393710 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.393675    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"
Oct 31 00:37:42.393710 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.393695    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" event={"ID":"f1f5b040-977a-47cc-95f7-c1ef0370e09a","Type":"ContainerDied","Data":"fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0"}
Oct 31 00:37:42.394189 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.393743    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd" event={"ID":"f1f5b040-977a-47cc-95f7-c1ef0370e09a","Type":"ContainerDied","Data":"1980df4576c4a8cdf959648f6d14d80f6597ab54915c4676f55d390d74248479"}
Oct 31 00:37:42.394189 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.393770    4888 scope.go:122] "RemoveContainer" containerID="fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0"
Oct 31 00:37:42.402391 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.402372    4888 scope.go:122] "RemoveContainer" containerID="fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0"
Oct 31 00:37:42.402650 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:37:42.402628    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0\": container with ID starting with fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0 not found: ID does not exist" containerID="fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0"
Oct 31 00:37:42.402697 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.402659    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0"} err="failed to get container status \"fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0\": rpc error: code = NotFound desc = could not find container \"fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0\": container with ID starting with fd8c05826855f2c2d6d64711e1a41f590eb9682321684f737c0cee7d3622a2a0 not found: ID does not exist"
Oct 31 00:37:42.554495 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.554460    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"]
Oct 31 00:37:42.560635 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:42.560619    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6mctd"]
Oct 31 00:37:43.552900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:43.552868    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" path="/var/lib/kubelet/pods/f1f5b040-977a-47cc-95f7-c1ef0370e09a/volumes"
Oct 31 00:37:48.551821 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:37:48.551787    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:37:54.166916 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.166876    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7696d678f8-p9gkj"]
Oct 31 00:37:54.167433 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.167395    4888 cpu_manager.go:414] "containerMap: removing container" podUID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" containerName="discovery"
Oct 31 00:37:54.167512 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.167436    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" containerName="discovery"
Oct 31 00:37:54.167549 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.167529    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f5b040-977a-47cc-95f7-c1ef0370e09a" containerName="discovery"
Oct 31 00:37:54.172212 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.172195    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.233569 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.233541    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7696d678f8-p9gkj"]
Oct 31 00:37:54.271123 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.271095    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pchq\" (UniqueName: \"kubernetes.io/projected/a3396fc1-5d90-4c2a-a482-f90fef453846-kube-api-access-9pchq\") pod \"llmisvc-controller-manager-7696d678f8-p9gkj\" (UID: \"a3396fc1-5d90-4c2a-a482-f90fef453846\") " pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.271252 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.271162    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3396fc1-5d90-4c2a-a482-f90fef453846-cert\") pod \"llmisvc-controller-manager-7696d678f8-p9gkj\" (UID: \"a3396fc1-5d90-4c2a-a482-f90fef453846\") " pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.280943 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.280923    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-lnhl4\""
Oct 31 00:37:54.281073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.281055    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Oct 31 00:37:54.281625 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.281612    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 31 00:37:54.326965 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.326944    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 31 00:37:54.372164 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.372137    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pchq\" (UniqueName: \"kubernetes.io/projected/a3396fc1-5d90-4c2a-a482-f90fef453846-kube-api-access-9pchq\") pod \"llmisvc-controller-manager-7696d678f8-p9gkj\" (UID: \"a3396fc1-5d90-4c2a-a482-f90fef453846\") " pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.372299 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.372184    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3396fc1-5d90-4c2a-a482-f90fef453846-cert\") pod \"llmisvc-controller-manager-7696d678f8-p9gkj\" (UID: \"a3396fc1-5d90-4c2a-a482-f90fef453846\") " pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.374699 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.374683    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3396fc1-5d90-4c2a-a482-f90fef453846-cert\") pod \"llmisvc-controller-manager-7696d678f8-p9gkj\" (UID: \"a3396fc1-5d90-4c2a-a482-f90fef453846\") " pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.431990 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.431931    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pchq\" (UniqueName: \"kubernetes.io/projected/a3396fc1-5d90-4c2a-a482-f90fef453846-kube-api-access-9pchq\") pod \"llmisvc-controller-manager-7696d678f8-p9gkj\" (UID: \"a3396fc1-5d90-4c2a-a482-f90fef453846\") " pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.485155 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.485121    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:37:54.850241 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:54.850213    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7696d678f8-p9gkj"]
Oct 31 00:37:54.856072 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:37:54.856042    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3396fc1_5d90_4c2a_a482_f90fef453846.slice/crio-631f9ff6d6e9c5cbdf6e988e2dede1bee0ac8975a482667e744587f81fce067c WatchSource:0}: Error finding container 631f9ff6d6e9c5cbdf6e988e2dede1bee0ac8975a482667e744587f81fce067c: Status 404 returned error can't find the container with id 631f9ff6d6e9c5cbdf6e988e2dede1bee0ac8975a482667e744587f81fce067c
Oct 31 00:37:55.452142 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:37:55.452112    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj" event={"ID":"a3396fc1-5d90-4c2a-a482-f90fef453846","Type":"ContainerStarted","Data":"631f9ff6d6e9c5cbdf6e988e2dede1bee0ac8975a482667e744587f81fce067c"}
Oct 31 00:38:02.488606 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:38:02.488575    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj" event={"ID":"a3396fc1-5d90-4c2a-a482-f90fef453846","Type":"ContainerStarted","Data":"c996df982e03b45a22f6df796c1fbd60f86c34cba2f1ee784b6138caf9541ab9"}
Oct 31 00:38:02.489000 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:38:02.488682    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:38:02.600885 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:38:02.600839    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj" podStartSLOduration=1.436686223 podStartE2EDuration="8.600822348s" totalImagesPullingTime="7.164136125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:02.600707109 +0000 UTC m=+491.155594547" watchObservedRunningTime="2026-10-31 00:38:02.600822348 +0000 UTC m=+491.155709786"
Oct 31 00:38:18.551492 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:38:18.551461    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:38:33.495467 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:38:33.495440    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7696d678f8-p9gkj"
Oct 31 00:38:48.552288 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:38:48.552256    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:39:18.550742 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:39:18.550702    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:39:48.551660 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:39:48.551624    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:39:51.606723 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:39:51.606695    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:39:51.607206 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:39:51.606962    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:40:18.551492 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:40:18.551458    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:40:48.552057 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:40:48.552023    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:41:18.551204 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:41:18.551129    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:41:48.561598 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:41:48.561562    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:41:55.514769 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:55.514737    4888 generic.go:356] "Generic (PLEG): container finished" podID="8d60c10e-c079-4f3b-a08e-843d51c9d457" containerID="c6c4320b742831ea521529f3716fa6fde9f77dcfd9306d838436abb99b4c8a9a" exitCode=0
Oct 31 00:41:55.515219 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:55.514818    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" event={"ID":"8d60c10e-c079-4f3b-a08e-843d51c9d457","Type":"ContainerDied","Data":"c6c4320b742831ea521529f3716fa6fde9f77dcfd9306d838436abb99b4c8a9a"}
Oct 31 00:41:55.515219 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:55.515156    4888 scope.go:122] "RemoveContainer" containerID="c6c4320b742831ea521529f3716fa6fde9f77dcfd9306d838436abb99b4c8a9a"
Oct 31 00:41:55.516130 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:55.516103    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:41:55.516185 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:55.516173    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:41:55.516227 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:55.516207    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:41:56.540574 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:56.540532    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" event={"ID":"8d60c10e-c079-4f3b-a08e-843d51c9d457","Type":"ContainerStarted","Data":"ac46e135d2e712485dddb9ff706dd901cbdc62951de4666e2fa4b905ce94aaa8"}
Oct 31 00:41:57.108899 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:57.108869    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:41:57.109198 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:57.109174    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xc7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Oct 31 00:41:57.109281 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:57.109258    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" podUID="8d60c10e-c079-4f3b-a08e-843d51c9d457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Oct 31 00:41:58.109396 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:58.109336    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xc7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Oct 31 00:41:58.109833 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:58.109449    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" podUID="8d60c10e-c079-4f3b-a08e-843d51c9d457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Oct 31 00:41:59.109728 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:59.109690    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xc7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Oct 31 00:41:59.110131 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:41:59.109745    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" podUID="8d60c10e-c079-4f3b-a08e-843d51c9d457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Oct 31 00:42:00.109677 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:42:00.109642    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xc7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Oct 31 00:42:00.109854 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:42:00.109695    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" podUID="8d60c10e-c079-4f3b-a08e-843d51c9d457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Oct 31 00:42:01.109774 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:42:01.109740    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xc7wn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused" start-of-body=
Oct 31 00:42:01.109946 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:42:01.109796    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn" podUID="8d60c10e-c079-4f3b-a08e-843d51c9d457" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.21:15021/healthz/ready\": dial tcp 10.134.0.21:15021: connect: connection refused"
Oct 31 00:42:02.113247 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:42:02.113219    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:42:02.113692 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:42:02.113606    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:42:02.114196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:42:02.114178    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xc7wn"
Oct 31 00:42:18.551814 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:42:18.551778    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:42:48.550723 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:42:48.550687    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:43:18.551192 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:43:18.551153    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:43:48.550608 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:43:48.550570    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:43:56.367654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.367626    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"]
Oct 31 00:43:56.376530 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.376507    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.399732 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.399707    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-model-cache\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.399871 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.399748    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-tmp-dir\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.399871 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.399820    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.399871 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.399844    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fcqc\" (UniqueName: \"kubernetes.io/projected/29022ae8-74d7-4720-b943-c5bdbccfc725-kube-api-access-4fcqc\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.400004 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.399874    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-dshm\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.400004 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.399911    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29022ae8-74d7-4720-b943-c5bdbccfc725-tls-certs\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.400004 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.399987    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-home\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.422476 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.422448    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"]
Oct 31 00:43:56.428532 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.428506    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Oct 31 00:43:56.467721 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.467694    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-sq55n\""
Oct 31 00:43:56.467721 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.467695    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Oct 31 00:43:56.467895 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.467720    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Oct 31 00:43:56.467895 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.467800    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Oct 31 00:43:56.500332 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500308    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-tmp-dir\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.500482 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500339    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.500482 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500471    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fcqc\" (UniqueName: \"kubernetes.io/projected/29022ae8-74d7-4720-b943-c5bdbccfc725-kube-api-access-4fcqc\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.500597 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500510    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-dshm\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.500597 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500533    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29022ae8-74d7-4720-b943-c5bdbccfc725-tls-certs\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.500702 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500595    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-home\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.500702 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500692    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-model-cache\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.500797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500758    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-tmp-dir\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.501014 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.500987    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-model-cache\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.501132 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.501031    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.501132 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.501042    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-home\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.502701 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.502682    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-dshm\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.503083 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.503064    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29022ae8-74d7-4720-b943-c5bdbccfc725-tls-certs\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.561550 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.561527    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fcqc\" (UniqueName: \"kubernetes.io/projected/29022ae8-74d7-4720-b943-c5bdbccfc725-kube-api-access-4fcqc\") pod \"cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:56.690646 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:56.690566    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:43:57.108525 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:57.108489    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"]
Oct 31 00:43:57.109881 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:43:57.109857    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29022ae8_74d7_4720_b943_c5bdbccfc725.slice/crio-cb8a9ae2e1949ff9c0ea3c8e26e758dd215137c0ab2068c691f0cf4623381d27 WatchSource:0}: Error finding container cb8a9ae2e1949ff9c0ea3c8e26e758dd215137c0ab2068c691f0cf4623381d27: Status 404 returned error can't find the container with id cb8a9ae2e1949ff9c0ea3c8e26e758dd215137c0ab2068c691f0cf4623381d27
Oct 31 00:43:57.111659 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:57.111646    4888 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:43:58.098688 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:43:58.098653    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" event={"ID":"29022ae8-74d7-4720-b943-c5bdbccfc725","Type":"ContainerStarted","Data":"cb8a9ae2e1949ff9c0ea3c8e26e758dd215137c0ab2068c691f0cf4623381d27"}
Oct 31 00:44:02.122119 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.122084    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" event={"ID":"29022ae8-74d7-4720-b943-c5bdbccfc725","Type":"ContainerStarted","Data":"14a75b086045f21ea4b73304cdb670dd2d1a56291903cdd72c5246c5ec986fd5"}
Oct 31 00:44:02.138854 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.138827    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"]
Oct 31 00:44:02.142236 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.142214    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.142845 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.142824    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.142943 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.142877    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.142988 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.142970    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2561c6e9-0985-4414-aa04-c4f5bbb68987-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.143035 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.143009    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.143112 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.143088    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgm2v\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-kube-api-access-wgm2v\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.143170 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.143138    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.143304 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.143281    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.143442 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.143336    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.143442 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.143409    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.243717 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.243685    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.243892 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.243733    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.243892 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.243860    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244037 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.243901    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244037 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.243944    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244037 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244003    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244037 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244028    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2561c6e9-0985-4414-aa04-c4f5bbb68987-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244251 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244078    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244251 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244108    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgm2v\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-kube-api-access-wgm2v\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244390 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244297    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244390 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244380    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244473 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244407    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244653 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244633    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.244978 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.244917    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2561c6e9-0985-4414-aa04-c4f5bbb68987-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.246400 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.246373    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.246630 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.246609    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.298867 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.298824    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"]
Oct 31 00:44:02.303063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.303037    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.303200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.303080    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgm2v\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-kube-api-access-wgm2v\") pod \"openshift-ai-inference-openshift-default-568c495856-p857s\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.457330 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.457301    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:02.874461 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.874428    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"]
Oct 31 00:44:02.876716 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:44:02.876687    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2561c6e9_0985_4414_aa04_c4f5bbb68987.slice/crio-05cfb876fd5870d0463baa7099dab403293b2168927cbeb983fc0f98fbc0639c WatchSource:0}: Error finding container 05cfb876fd5870d0463baa7099dab403293b2168927cbeb983fc0f98fbc0639c: Status 404 returned error can't find the container with id 05cfb876fd5870d0463baa7099dab403293b2168927cbeb983fc0f98fbc0639c
Oct 31 00:44:02.878868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.878835    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:44:02.878935 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.878897    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:44:02.878935 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:02.878925    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:44:03.129952 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:03.129842    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" event={"ID":"2561c6e9-0985-4414-aa04-c4f5bbb68987","Type":"ContainerStarted","Data":"bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975"}
Oct 31 00:44:03.129952 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:03.129929    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" event={"ID":"2561c6e9-0985-4414-aa04-c4f5bbb68987","Type":"ContainerStarted","Data":"05cfb876fd5870d0463baa7099dab403293b2168927cbeb983fc0f98fbc0639c"}
Oct 31 00:44:03.248843 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:03.248771    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" podStartSLOduration=1.248754532 podStartE2EDuration="1.248754532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:03.248317808 +0000 UTC m=+851.803205247" watchObservedRunningTime="2026-10-31 00:44:03.248754532 +0000 UTC m=+851.803641972"
Oct 31 00:44:03.458547 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:03.458514    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:03.464990 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:03.464948    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:04.135409 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:04.135378    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:04.136367 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:04.136333    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:44:07.150781 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:07.150751    4888 generic.go:356] "Generic (PLEG): container finished" podID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerID="14a75b086045f21ea4b73304cdb670dd2d1a56291903cdd72c5246c5ec986fd5" exitCode=0
Oct 31 00:44:07.151180 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:07.150843    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" event={"ID":"29022ae8-74d7-4720-b943-c5bdbccfc725","Type":"ContainerDied","Data":"14a75b086045f21ea4b73304cdb670dd2d1a56291903cdd72c5246c5ec986fd5"}
Oct 31 00:44:07.296849 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:07.296822    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"]
Oct 31 00:44:17.268673 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.268637    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"]
Oct 31 00:44:17.278155 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.278128    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.378111 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.378079    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"]
Oct 31 00:44:17.379029 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.378995    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379158 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379079    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379158 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379109    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cae128-1e0e-4709-aeed-14737f1f9601-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379277 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379195    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379277 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379239    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379376 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379309    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379343    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnlx4\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-kube-api-access-dnlx4\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379418 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379406    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.379491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.379432    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480183    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480236    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnlx4\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-kube-api-access-dnlx4\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480266    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480293    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480340    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480436    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480463    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cae128-1e0e-4709-aeed-14737f1f9601-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480727 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480489    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480727 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480524    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480727 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480682    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.480888 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.480863    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.481069 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.481050    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.481343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.481301    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cae128-1e0e-4709-aeed-14737f1f9601-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.481568 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.481397    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.484013 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.483991    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.484176 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.484162    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.541097 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.541026    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnlx4\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-kube-api-access-dnlx4\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.541228 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.541202    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-m76bd\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:17.594304 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:17.594089    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:18.067114 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.067083    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"]
Oct 31 00:44:18.070241 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:44:18.070208    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3cae128_1e0e_4709_aeed_14737f1f9601.slice/crio-9d2a66780c5673c7884904ab7e6894611ef79f0fb2f91f47384004240ec12ed7 WatchSource:0}: Error finding container 9d2a66780c5673c7884904ab7e6894611ef79f0fb2f91f47384004240ec12ed7: Status 404 returned error can't find the container with id 9d2a66780c5673c7884904ab7e6894611ef79f0fb2f91f47384004240ec12ed7
Oct 31 00:44:18.073092 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.072660    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:44:18.073092 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.072763    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:44:18.073092 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.072796    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:44:18.222105 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.221278    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" event={"ID":"f3cae128-1e0e-4709-aeed-14737f1f9601","Type":"ContainerStarted","Data":"f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8"}
Oct 31 00:44:18.222105 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.221374    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" event={"ID":"f3cae128-1e0e-4709-aeed-14737f1f9601","Type":"ContainerStarted","Data":"9d2a66780c5673c7884904ab7e6894611ef79f0fb2f91f47384004240ec12ed7"}
Oct 31 00:44:18.342154 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.341841    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podStartSLOduration=1.341825442 podStartE2EDuration="1.341825442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:18.337392167 +0000 UTC m=+866.892279603" watchObservedRunningTime="2026-10-31 00:44:18.341825442 +0000 UTC m=+866.896712880"
Oct 31 00:44:18.564844 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:44:18.564811    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:44:18.594626 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:18.594553    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:19.601182 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:19.601146    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:20.233211 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:20.233078    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:20.234151 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:20.234118    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:44:36.315265 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:36.315232    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" event={"ID":"29022ae8-74d7-4720-b943-c5bdbccfc725","Type":"ContainerStarted","Data":"e4a88b0dc6b195f9204a09d0ae6bdd7a2893fd3bbf2fe1fd5c7ad2f63d46bd2e"}
Oct 31 00:44:36.527961 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:36.527889    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" podStartSLOduration=2.664578212 podStartE2EDuration="40.527871115s" totalImagesPullingTime="33.148988218s" totalInitContainerRuntime="4.714304685s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:36.526091929 +0000 UTC m=+885.080979364" watchObservedRunningTime="2026-10-31 00:44:36.527871115 +0000 UTC m=+885.082758740"
Oct 31 00:44:36.691144 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:36.691046    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:44:48.553591 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:44:48.553554    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:44:51.337246 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:51.337205    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" podUID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerName="main" containerID="cri-o://e4a88b0dc6b195f9204a09d0ae6bdd7a2893fd3bbf2fe1fd5c7ad2f63d46bd2e" gracePeriod=45
Oct 31 00:44:51.643786 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:51.643698    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:44:51.644576 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:44:51.644553    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:45:06.273569 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:06.273523    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-m76bd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:45:06.274120 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:06.273608    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:45:08.537756 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.537671    4888 generic.go:356] "Generic (PLEG): container finished" podID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerID="f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8" exitCode=0
Oct 31 00:45:08.537756 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.537748    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" event={"ID":"f3cae128-1e0e-4709-aeed-14737f1f9601","Type":"ContainerDied","Data":"f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8"}
Oct 31 00:45:08.538322 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.538152    4888 scope.go:122] "RemoveContainer" containerID="f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8"
Oct 31 00:45:08.539940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.539435    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:45:08.539940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.539533    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:45:08.539940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.539580    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:45:08.539940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.539611    4888 generic.go:356] "Generic (PLEG): container finished" podID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerID="bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975" exitCode=0
Oct 31 00:45:08.539940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.539647    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" event={"ID":"2561c6e9-0985-4414-aa04-c4f5bbb68987","Type":"ContainerDied","Data":"bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975"}
Oct 31 00:45:08.539940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.539905    4888 scope.go:122] "RemoveContainer" containerID="bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975"
Oct 31 00:45:08.540922 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.540896    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:45:08.541018 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.540954    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:45:08.541018 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:08.540990    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:45:09.570375 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:09.569287    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" event={"ID":"f3cae128-1e0e-4709-aeed-14737f1f9601","Type":"ContainerStarted","Data":"24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086"}
Oct 31 00:45:09.573393 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:09.573278    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" event={"ID":"2561c6e9-0985-4414-aa04-c4f5bbb68987","Type":"ContainerStarted","Data":"430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d"}
Oct 31 00:45:09.594772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:09.594742    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:45:09.595003 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:09.594980    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-m76bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 00:45:09.595098 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:09.595033    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 00:45:10.460377 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:10.459773    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:45:10.460377 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:10.459897    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-p857s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 00:45:10.460377 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:10.459943    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 00:45:10.594648 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:10.594616    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-m76bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 00:45:10.595026 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:10.594676    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 00:45:11.461816 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:11.461772    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-p857s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 00:45:11.462095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:11.462064    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 00:45:11.594787 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:11.594752    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-m76bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 00:45:11.595186 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:11.594811    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 00:45:12.457796 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:12.457757    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-p857s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 00:45:12.458097 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:12.458067    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 00:45:12.595423 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:12.595395    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-m76bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 00:45:12.595814 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:12.595449    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 00:45:13.460375 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:13.458744    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-p857s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 00:45:13.460375 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:13.458816    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 00:45:13.595439 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:13.595411    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-m76bd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 00:45:13.595876 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:13.595854    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 00:45:14.458117 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:14.458081    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-p857s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 00:45:14.458300 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:14.458138    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 00:45:14.624864 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:14.624824    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:45:14.625503 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:14.625477    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:45:14.628103 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:14.628078    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:45:15.462605 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:15.462574    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:45:15.463013 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:15.462988    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:45:15.463658 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:15.463639    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:45:18.552877 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:45:18.552840    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:45:36.356524 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:45:36.356494    4888 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-11794-0.service\": RecentStats: unable to find data in memory cache]"
Oct 31 00:45:36.357132 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:45:36.356650    4888 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@2-11794-0.service\": RecentStats: unable to find data in memory cache], [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@1-11793-0.service\": RecentStats: unable to find data in memory cache]"
Oct 31 00:45:36.708977 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:36.708947    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5_29022ae8-74d7-4720-b943-c5bdbccfc725/main/0.log"
Oct 31 00:45:36.709318 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:36.709294    4888 generic.go:356] "Generic (PLEG): container finished" podID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerID="e4a88b0dc6b195f9204a09d0ae6bdd7a2893fd3bbf2fe1fd5c7ad2f63d46bd2e" exitCode=137
Oct 31 00:45:36.709402 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:36.709381    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" event={"ID":"29022ae8-74d7-4720-b943-c5bdbccfc725","Type":"ContainerDied","Data":"e4a88b0dc6b195f9204a09d0ae6bdd7a2893fd3bbf2fe1fd5c7ad2f63d46bd2e"}
Oct 31 00:45:36.980913 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:36.980886    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5_29022ae8-74d7-4720-b943-c5bdbccfc725/main/0.log"
Oct 31 00:45:36.982261 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:36.982242    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:45:37.073985 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.073955    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-model-cache\") pod \"29022ae8-74d7-4720-b943-c5bdbccfc725\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") "
Oct 31 00:45:37.074153 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074029    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-tmp-dir\") pod \"29022ae8-74d7-4720-b943-c5bdbccfc725\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") "
Oct 31 00:45:37.074153 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074080    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-dshm\") pod \"29022ae8-74d7-4720-b943-c5bdbccfc725\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") "
Oct 31 00:45:37.074153 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074106    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29022ae8-74d7-4720-b943-c5bdbccfc725-tls-certs\" (UniqueName: \"kubernetes.io/secret/29022ae8-74d7-4720-b943-c5bdbccfc725-tls-certs\") pod \"29022ae8-74d7-4720-b943-c5bdbccfc725\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") "
Oct 31 00:45:37.074153 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074135    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29022ae8-74d7-4720-b943-c5bdbccfc725-kube-api-access-4fcqc\" (UniqueName: \"kubernetes.io/projected/29022ae8-74d7-4720-b943-c5bdbccfc725-kube-api-access-4fcqc\") pod \"29022ae8-74d7-4720-b943-c5bdbccfc725\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") "
Oct 31 00:45:37.074398 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074166    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-kserve-provision-location\") pod \"29022ae8-74d7-4720-b943-c5bdbccfc725\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") "
Oct 31 00:45:37.074398 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074208    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-home\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-home\") pod \"29022ae8-74d7-4720-b943-c5bdbccfc725\" (UID: \"29022ae8-74d7-4720-b943-c5bdbccfc725\") "
Oct 31 00:45:37.074398 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074232    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-model-cache" pod "29022ae8-74d7-4720-b943-c5bdbccfc725" (UID: "29022ae8-74d7-4720-b943-c5bdbccfc725"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:37.074559 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074454    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:37.074664 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074643    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-home" pod "29022ae8-74d7-4720-b943-c5bdbccfc725" (UID: "29022ae8-74d7-4720-b943-c5bdbccfc725"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:37.074737 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.074657    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-tmp-dir" pod "29022ae8-74d7-4720-b943-c5bdbccfc725" (UID: "29022ae8-74d7-4720-b943-c5bdbccfc725"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:37.076521 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.076496    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-dshm" pod "29022ae8-74d7-4720-b943-c5bdbccfc725" (UID: "29022ae8-74d7-4720-b943-c5bdbccfc725"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:37.076616 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.076524    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29022ae8-74d7-4720-b943-c5bdbccfc725-tls-certs" pod "29022ae8-74d7-4720-b943-c5bdbccfc725" (UID: "29022ae8-74d7-4720-b943-c5bdbccfc725"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:45:37.076727 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.076710    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29022ae8-74d7-4720-b943-c5bdbccfc725-kube-api-access-4fcqc" pod "29022ae8-74d7-4720-b943-c5bdbccfc725" (UID: "29022ae8-74d7-4720-b943-c5bdbccfc725"). InnerVolumeSpecName "kube-api-access-4fcqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:37.126474 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.126436    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-kserve-provision-location" pod "29022ae8-74d7-4720-b943-c5bdbccfc725" (UID: "29022ae8-74d7-4720-b943-c5bdbccfc725"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:37.175596 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.175572    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:37.175596 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.175594    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29022ae8-74d7-4720-b943-c5bdbccfc725-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:37.175752 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.175604    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fcqc\" (UniqueName: \"kubernetes.io/projected/29022ae8-74d7-4720-b943-c5bdbccfc725-kube-api-access-4fcqc\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:37.175752 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.175613    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:37.175752 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.175621    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:37.175752 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.175648    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29022ae8-74d7-4720-b943-c5bdbccfc725-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:45:37.715164 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.715137    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5_29022ae8-74d7-4720-b943-c5bdbccfc725/main/0.log"
Oct 31 00:45:37.715594 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.715577    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"
Oct 31 00:45:37.715673 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.715577    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5" event={"ID":"29022ae8-74d7-4720-b943-c5bdbccfc725","Type":"ContainerDied","Data":"cb8a9ae2e1949ff9c0ea3c8e26e758dd215137c0ab2068c691f0cf4623381d27"}
Oct 31 00:45:37.715714 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.715699    4888 scope.go:122] "RemoveContainer" containerID="e4a88b0dc6b195f9204a09d0ae6bdd7a2893fd3bbf2fe1fd5c7ad2f63d46bd2e"
Oct 31 00:45:37.724049 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.724024    4888 scope.go:122] "RemoveContainer" containerID="14a75b086045f21ea4b73304cdb670dd2d1a56291903cdd72c5246c5ec986fd5"
Oct 31 00:45:37.924273 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.924191    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"]
Oct 31 00:45:37.973284 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:37.973252    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-5fd5bdf68f-hzjf5"]
Oct 31 00:45:39.553581 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:39.553551    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29022ae8-74d7-4720-b943-c5bdbccfc725" path="/var/lib/kubelet/pods/29022ae8-74d7-4720-b943-c5bdbccfc725/volumes"
Oct 31 00:45:48.551413 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:45:48.551381    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:45:49.238404 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.238368    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"]
Oct 31 00:45:49.238747 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.238730    4888 cpu_manager.go:414] "containerMap: removing container" podUID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerName="storage-initializer"
Oct 31 00:45:49.238840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.238749    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerName="storage-initializer"
Oct 31 00:45:49.238840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.238769    4888 cpu_manager.go:414] "containerMap: removing container" podUID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerName="main"
Oct 31 00:45:49.238840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.238778    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerName="main"
Oct 31 00:45:49.238990 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.238890    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="29022ae8-74d7-4720-b943-c5bdbccfc725" containerName="main"
Oct 31 00:45:49.241762 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.241741    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.305258 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.305231    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Oct 31 00:45:49.339453 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.339421    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cpu-accelerator-preset-ser-afc820a4-epp-sa-dockercfg-5mxx8\""
Oct 31 00:45:49.339453 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.339439    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Oct 31 00:45:49.339683 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.339478    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-64tdh\""
Oct 31 00:45:49.339683 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.339521    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Oct 31 00:45:49.341232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.341211    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"]
Oct 31 00:45:49.376400 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.376362    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cnqm\" (UniqueName: \"kubernetes.io/projected/d88cad37-83ac-43e7-a3f3-82941b612dd3-kube-api-access-2cnqm\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.376578 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.376552    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88cad37-83ac-43e7-a3f3-82941b612dd3-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.477574 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.477501    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88cad37-83ac-43e7-a3f3-82941b612dd3-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.477574 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.477544    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnqm\" (UniqueName: \"kubernetes.io/projected/d88cad37-83ac-43e7-a3f3-82941b612dd3-kube-api-access-2cnqm\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.480027 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.480007    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88cad37-83ac-43e7-a3f3-82941b612dd3-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.537386 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.537340    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cnqm\" (UniqueName: \"kubernetes.io/projected/d88cad37-83ac-43e7-a3f3-82941b612dd3-kube-api-access-2cnqm\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.553469 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.553447    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:49.920193 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:49.920171    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"]
Oct 31 00:45:49.922304 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:45:49.922280    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88cad37_83ac_43e7_a3f3_82941b612dd3.slice/crio-8b67ad2933e453a15213a3f959865ecbdef2d7825c0a8b948a5ca3c66c0b2a1d WatchSource:0}: Error finding container 8b67ad2933e453a15213a3f959865ecbdef2d7825c0a8b948a5ca3c66c0b2a1d: Status 404 returned error can't find the container with id 8b67ad2933e453a15213a3f959865ecbdef2d7825c0a8b948a5ca3c66c0b2a1d
Oct 31 00:45:50.778940 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:50.778908    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" event={"ID":"d88cad37-83ac-43e7-a3f3-82941b612dd3","Type":"ContainerStarted","Data":"8b67ad2933e453a15213a3f959865ecbdef2d7825c0a8b948a5ca3c66c0b2a1d"}
Oct 31 00:45:52.789523 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:52.789481    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" event={"ID":"d88cad37-83ac-43e7-a3f3-82941b612dd3","Type":"ContainerStarted","Data":"a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41"}
Oct 31 00:45:52.789952 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:52.789593    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:45:52.904029 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:45:52.903978    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" podStartSLOduration=1.8173516360000002 podStartE2EDuration="3.903963409s" totalImagesPullingTime="2.086611773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:52.9021106 +0000 UTC m=+961.456998034" watchObservedRunningTime="2026-10-31 00:45:52.903963409 +0000 UTC m=+961.458850846"
Oct 31 00:46:18.551621 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:46:18.551584    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:46:23.796738 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:23.796712    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:46:28.702361 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.702327    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"]
Oct 31 00:46:28.705744 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.705728    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:28.756719 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.756693    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Oct 31 00:46:28.756856 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.756758    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Oct 31 00:46:28.801571 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.801545    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76574633-8513-4177-94dc-b4bdce820273-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:28.801720 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.801604    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjv6t\" (UniqueName: \"kubernetes.io/projected/76574633-8513-4177-94dc-b4bdce820273-kube-api-access-vjv6t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:28.804199 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.804179    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"]
Oct 31 00:46:28.837748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.837718    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-svz8k\""
Oct 31 00:46:28.837748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.837730    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-l6rcl\""
Oct 31 00:46:28.837748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.837729    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Oct 31 00:46:28.902375 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.902316    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjv6t\" (UniqueName: \"kubernetes.io/projected/76574633-8513-4177-94dc-b4bdce820273-kube-api-access-vjv6t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:28.902561 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.902426    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76574633-8513-4177-94dc-b4bdce820273-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:28.904946 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.904922    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76574633-8513-4177-94dc-b4bdce820273-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:28.961874 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:28.961802    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjv6t\" (UniqueName: \"kubernetes.io/projected/76574633-8513-4177-94dc-b4bdce820273-kube-api-access-vjv6t\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:29.018144 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:29.018109    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:29.384785 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:29.384742    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"]
Oct 31 00:46:29.389156 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:46:29.389123    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76574633_8513_4177_94dc_b4bdce820273.slice/crio-0daa5b03cffe2609ed69e8c7f9d1bc0d8bf30b7d76fe8dccbda741409dbfed25 WatchSource:0}: Error finding container 0daa5b03cffe2609ed69e8c7f9d1bc0d8bf30b7d76fe8dccbda741409dbfed25: Status 404 returned error can't find the container with id 0daa5b03cffe2609ed69e8c7f9d1bc0d8bf30b7d76fe8dccbda741409dbfed25
Oct 31 00:46:29.963464 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:29.963433    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" event={"ID":"76574633-8513-4177-94dc-b4bdce820273","Type":"ContainerStarted","Data":"f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94"}
Oct 31 00:46:29.963882 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:29.963480    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" event={"ID":"76574633-8513-4177-94dc-b4bdce820273","Type":"ContainerStarted","Data":"0daa5b03cffe2609ed69e8c7f9d1bc0d8bf30b7d76fe8dccbda741409dbfed25"}
Oct 31 00:46:29.963882 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:29.963546    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:46:30.075971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:46:30.075921    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podStartSLOduration=2.075908473 podStartE2EDuration="2.075908473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:30.075252614 +0000 UTC m=+998.630140076" watchObservedRunningTime="2026-10-31 00:46:30.075908473 +0000 UTC m=+998.630795910"
Oct 31 00:46:48.551958 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:46:48.551921    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:47:00.969999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:00.969914    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:07.773599 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:07.773564    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"]
Oct 31 00:47:07.775477 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:07.775441    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:17.775666 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:17.775621    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:18.552014 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:47:18.551977    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:47:20.368766 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:20.368735    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"]
Oct 31 00:47:22.793284 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:22.793215    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main" containerID="cri-o://f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94" gracePeriod=45
Oct 31 00:47:23.797105 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:23.797069    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:27.775652 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:27.775610    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:33.796301 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:33.796266    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:35.388547 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:35.388508    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main" containerID="cri-o://a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41" gracePeriod=45
Oct 31 00:47:37.775621 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:37.775586    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:43.797759 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:43.797721    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:43.798238 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:43.798117    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:47:47.775126 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:47.775090    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:47:48.551121 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:47:48.551085    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:47:53.033334 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.033311    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:47:53.111900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.111830    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76574633-8513-4177-94dc-b4bdce820273-kube-api-access-vjv6t\" (UniqueName: \"kubernetes.io/projected/76574633-8513-4177-94dc-b4bdce820273-kube-api-access-vjv6t\") pod \"76574633-8513-4177-94dc-b4bdce820273\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") "
Oct 31 00:47:53.111900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.111893    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76574633-8513-4177-94dc-b4bdce820273-tls-certs\" (UniqueName: \"kubernetes.io/secret/76574633-8513-4177-94dc-b4bdce820273-tls-certs\") pod \"76574633-8513-4177-94dc-b4bdce820273\" (UID: \"76574633-8513-4177-94dc-b4bdce820273\") "
Oct 31 00:47:53.114024 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.113992    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76574633-8513-4177-94dc-b4bdce820273-tls-certs" pod "76574633-8513-4177-94dc-b4bdce820273" (UID: "76574633-8513-4177-94dc-b4bdce820273"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:47:53.114024 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.114010    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76574633-8513-4177-94dc-b4bdce820273-kube-api-access-vjv6t" pod "76574633-8513-4177-94dc-b4bdce820273" (UID: "76574633-8513-4177-94dc-b4bdce820273"). InnerVolumeSpecName "kube-api-access-vjv6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:47:53.213503 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.213451    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjv6t\" (UniqueName: \"kubernetes.io/projected/76574633-8513-4177-94dc-b4bdce820273-kube-api-access-vjv6t\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:53.213503 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.213501    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76574633-8513-4177-94dc-b4bdce820273-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:47:53.352674 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.352639    4888 generic.go:356] "Generic (PLEG): container finished" podID="76574633-8513-4177-94dc-b4bdce820273" containerID="f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94" exitCode=0
Oct 31 00:47:53.352834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.352692    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"
Oct 31 00:47:53.352834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.352717    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" event={"ID":"76574633-8513-4177-94dc-b4bdce820273","Type":"ContainerDied","Data":"f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94"}
Oct 31 00:47:53.352834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.352768    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp" event={"ID":"76574633-8513-4177-94dc-b4bdce820273","Type":"ContainerDied","Data":"0daa5b03cffe2609ed69e8c7f9d1bc0d8bf30b7d76fe8dccbda741409dbfed25"}
Oct 31 00:47:53.352834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.352799    4888 scope.go:122] "RemoveContainer" containerID="f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94"
Oct 31 00:47:53.361589 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.361573    4888 scope.go:122] "RemoveContainer" containerID="f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94"
Oct 31 00:47:53.361853 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:47:53.361832    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94\": container with ID starting with f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94 not found: ID does not exist" containerID="f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94"
Oct 31 00:47:53.361938 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.361864    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94"} err="failed to get container status \"f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94\": rpc error: code = NotFound desc = could not find container \"f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94\": container with ID starting with f0709b72e30eaf69e9f57c107cd0bea1ec84708a1a9440514a99e4039be48b94 not found: ID does not exist"
Oct 31 00:47:53.517017 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.516985    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"]
Oct 31 00:47:53.522965 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.522940    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b6xl5pp"]
Oct 31 00:47:53.553851 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.553813    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76574633-8513-4177-94dc-b4bdce820273" path="/var/lib/kubelet/pods/76574633-8513-4177-94dc-b4bdce820273/volumes"
Oct 31 00:47:53.796933 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:47:53.796898    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:48:03.797097 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:03.797059    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:48:05.632999 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:05.632974    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:48:05.714810 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:05.714787    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88cad37-83ac-43e7-a3f3-82941b612dd3-kube-api-access-2cnqm\" (UniqueName: \"kubernetes.io/projected/d88cad37-83ac-43e7-a3f3-82941b612dd3-kube-api-access-2cnqm\") pod \"d88cad37-83ac-43e7-a3f3-82941b612dd3\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") "
Oct 31 00:48:05.714992 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:05.714840    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d88cad37-83ac-43e7-a3f3-82941b612dd3-tls-certs\" (UniqueName: \"kubernetes.io/secret/d88cad37-83ac-43e7-a3f3-82941b612dd3-tls-certs\") pod \"d88cad37-83ac-43e7-a3f3-82941b612dd3\" (UID: \"d88cad37-83ac-43e7-a3f3-82941b612dd3\") "
Oct 31 00:48:05.717111 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:05.717085    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88cad37-83ac-43e7-a3f3-82941b612dd3-tls-certs" pod "d88cad37-83ac-43e7-a3f3-82941b612dd3" (UID: "d88cad37-83ac-43e7-a3f3-82941b612dd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:05.717224 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:05.717203    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88cad37-83ac-43e7-a3f3-82941b612dd3-kube-api-access-2cnqm" pod "d88cad37-83ac-43e7-a3f3-82941b612dd3" (UID: "d88cad37-83ac-43e7-a3f3-82941b612dd3"). InnerVolumeSpecName "kube-api-access-2cnqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:05.815662 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:05.815632    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cnqm\" (UniqueName: \"kubernetes.io/projected/d88cad37-83ac-43e7-a3f3-82941b612dd3-kube-api-access-2cnqm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:05.815662 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:05.815657    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88cad37-83ac-43e7-a3f3-82941b612dd3-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:48:06.416525 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.416494    4888 generic.go:356] "Generic (PLEG): container finished" podID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerID="a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41" exitCode=0
Oct 31 00:48:06.416686 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.416545    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"
Oct 31 00:48:06.416686 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.416584    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" event={"ID":"d88cad37-83ac-43e7-a3f3-82941b612dd3","Type":"ContainerDied","Data":"a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41"}
Oct 31 00:48:06.416686 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.416641    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p" event={"ID":"d88cad37-83ac-43e7-a3f3-82941b612dd3","Type":"ContainerDied","Data":"8b67ad2933e453a15213a3f959865ecbdef2d7825c0a8b948a5ca3c66c0b2a1d"}
Oct 31 00:48:06.416686 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.416669    4888 scope.go:122] "RemoveContainer" containerID="a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41"
Oct 31 00:48:06.425395 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.425379    4888 scope.go:122] "RemoveContainer" containerID="a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41"
Oct 31 00:48:06.425642 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:48:06.425624    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41\": container with ID starting with a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41 not found: ID does not exist" containerID="a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41"
Oct 31 00:48:06.425686 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.425651    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41"} err="failed to get container status \"a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41\": rpc error: code = NotFound desc = could not find container \"a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41\": container with ID starting with a763e8eb99d2adafb6b8e7664f13cb3087b0babc7faa55bafa64c2230335ae41 not found: ID does not exist"
Oct 31 00:48:06.575906 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.575875    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"]
Oct 31 00:48:06.622604 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:06.622579    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-sche5wg4p"]
Oct 31 00:48:07.554029 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:07.553994    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" path="/var/lib/kubelet/pods/d88cad37-83ac-43e7-a3f3-82941b612dd3/volumes"
Oct 31 00:48:18.551618 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:48:18.551576    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:48:19.142530 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.142497    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"]
Oct 31 00:48:19.142962 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.142947    4888 cpu_manager.go:414] "containerMap: removing container" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main"
Oct 31 00:48:19.143031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.142964    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main"
Oct 31 00:48:19.143031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.142984    4888 cpu_manager.go:414] "containerMap: removing container" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main"
Oct 31 00:48:19.143031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.142991    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main"
Oct 31 00:48:19.143138 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.143070    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="76574633-8513-4177-94dc-b4bdce820273" containerName="main"
Oct 31 00:48:19.143138 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.143080    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88cad37-83ac-43e7-a3f3-82941b612dd3" containerName="main"
Oct 31 00:48:19.147641 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.147622    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.237731 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.237698    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"]
Oct 31 00:48:19.299045 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299014    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vm5x\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-kube-api-access-6vm5x\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299045 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299047    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299243 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299065    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299243 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299081    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299243 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299218    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299337 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299261    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299337 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299291    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299337 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299315    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.299451 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.299438    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400447 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400371    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400447 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400415    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vm5x\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-kube-api-access-6vm5x\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400447 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400432    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400711 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400449    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400711 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400472    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400711 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400508    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400711 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400538    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400711 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400576    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.400711 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400606    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.401027 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400796    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.401027 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400906    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.401027 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.400938    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.401133 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.401114    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.401246 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.401229    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.402709 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.402686    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.403102 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.403082    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.460516 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.460491    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.460705 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.460685    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vm5x\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-kube-api-access-6vm5x\") pod \"openshift-ai-inference-openshift-default-568c495856-gxtp4\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:19.759949 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:19.759917    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:20.182516 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.182480    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"]
Oct 31 00:48:20.184694 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:48:20.184667    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab69a17d_f70d_467f_b211_15eb5fd1b05b.slice/crio-f46d2c632c14e0e139a33332790b5fca365d19b06df609eabc0570067399aba8 WatchSource:0}: Error finding container f46d2c632c14e0e139a33332790b5fca365d19b06df609eabc0570067399aba8: Status 404 returned error can't find the container with id f46d2c632c14e0e139a33332790b5fca365d19b06df609eabc0570067399aba8
Oct 31 00:48:20.186563 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.186535    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:48:20.186628 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.186592    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:48:20.186628 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.186620    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:48:20.562502 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.562462    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" event={"ID":"ab69a17d-f70d-467f-b211-15eb5fd1b05b","Type":"ContainerStarted","Data":"0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316"}
Oct 31 00:48:20.562675 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.562523    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" event={"ID":"ab69a17d-f70d-467f-b211-15eb5fd1b05b","Type":"ContainerStarted","Data":"f46d2c632c14e0e139a33332790b5fca365d19b06df609eabc0570067399aba8"}
Oct 31 00:48:20.684584 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.684529    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podStartSLOduration=1.684512519 podStartE2EDuration="1.684512519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:20.68311006 +0000 UTC m=+1109.237997502" watchObservedRunningTime="2026-10-31 00:48:20.684512519 +0000 UTC m=+1109.239399956"
Oct 31 00:48:20.760886 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:20.760834    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:21.476307 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.476272    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"]
Oct 31 00:48:21.479976 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.479958    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.535373 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.534393    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Oct 31 00:48:21.580554 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.580518    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Oct 31 00:48:21.580707 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.580615    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-wbkcp\""
Oct 31 00:48:21.580747 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.580706    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Oct 31 00:48:21.582504 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.582478    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"]
Oct 31 00:48:21.640772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.640736    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.640953 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.640798    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.640953 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.640863    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.640953 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.640888    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.640953 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.640915    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wllk\" (UniqueName: \"kubernetes.io/projected/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-kube-api-access-8wllk\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.641103 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.641004    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741473 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741385    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741473 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741428    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741473 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741443    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741473 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741475    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wllk\" (UniqueName: \"kubernetes.io/projected/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-kube-api-access-8wllk\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741506    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741585    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741934 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741849    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.741989 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741972    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.742082 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.741986    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.743929 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.743908    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.744083 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.744067    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:21.762103 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.762073    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-gxtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:48:21.762483 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.762141    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 31 00:48:21.801297 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:21.801274    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wllk\" (UniqueName: \"kubernetes.io/projected/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-kube-api-access-8wllk\") pod \"fc-smoke-test-kserve-7d8f9d9664-f6ghn\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:22.094013 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.093926    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:22.107394 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.107368    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"]
Oct 31 00:48:22.110834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.110807    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.162934 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.162907    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-k84wr\""
Oct 31 00:48:22.167200 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.167172    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"]
Oct 31 00:48:22.245949 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.245919    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1040ba98-a930-44d1-986c-bbba87fbbb5e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.246111 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.245979    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5zn\" (UniqueName: \"kubernetes.io/projected/1040ba98-a930-44d1-986c-bbba87fbbb5e-kube-api-access-jw5zn\") pod \"fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.347036 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.346949    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1040ba98-a930-44d1-986c-bbba87fbbb5e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.347036 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.346985    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5zn\" (UniqueName: \"kubernetes.io/projected/1040ba98-a930-44d1-986c-bbba87fbbb5e-kube-api-access-jw5zn\") pod \"fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.349793 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.349768    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1040ba98-a930-44d1-986c-bbba87fbbb5e-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.407037 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.407012    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5zn\" (UniqueName: \"kubernetes.io/projected/1040ba98-a930-44d1-986c-bbba87fbbb5e-kube-api-access-jw5zn\") pod \"fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.448935 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.448900    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:22.495275 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.495216    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"]
Oct 31 00:48:22.500477 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:48:22.498422    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f4515c_7e9f_4f69_a0c2_39bca2e5b10d.slice/crio-e98e757112cbc825c009692871d57604d8caf62fe6161f6b03213e17dc9169c2 WatchSource:0}: Error finding container e98e757112cbc825c009692871d57604d8caf62fe6161f6b03213e17dc9169c2: Status 404 returned error can't find the container with id e98e757112cbc825c009692871d57604d8caf62fe6161f6b03213e17dc9169c2
Oct 31 00:48:22.573887 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.573857    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" event={"ID":"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d","Type":"ContainerStarted","Data":"e98e757112cbc825c009692871d57604d8caf62fe6161f6b03213e17dc9169c2"}
Oct 31 00:48:22.760895 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.760864    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-gxtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:48:22.761076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.760931    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 31 00:48:22.838923 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:48:22.838479    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1040ba98_a930_44d1_986c_bbba87fbbb5e.slice/crio-9ce0e8d6dcc66f7573f7d2fb4e9c302719b32c1484f70581561787f1886c0c90 WatchSource:0}: Error finding container 9ce0e8d6dcc66f7573f7d2fb4e9c302719b32c1484f70581561787f1886c0c90: Status 404 returned error can't find the container with id 9ce0e8d6dcc66f7573f7d2fb4e9c302719b32c1484f70581561787f1886c0c90
Oct 31 00:48:22.839316 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:22.839140    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"]
Oct 31 00:48:23.582850 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:23.582805    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" event={"ID":"1040ba98-a930-44d1-986c-bbba87fbbb5e","Type":"ContainerStarted","Data":"e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14"}
Oct 31 00:48:23.583027 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:23.582865    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" event={"ID":"1040ba98-a930-44d1-986c-bbba87fbbb5e","Type":"ContainerStarted","Data":"9ce0e8d6dcc66f7573f7d2fb4e9c302719b32c1484f70581561787f1886c0c90"}
Oct 31 00:48:23.583027 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:23.582905    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:48:23.707174 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:23.707112    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" podStartSLOduration=1.707092099 podStartE2EDuration="1.707092099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:23.699681549 +0000 UTC m=+1112.254568989" watchObservedRunningTime="2026-10-31 00:48:23.707092099 +0000 UTC m=+1112.261979541"
Oct 31 00:48:23.760987 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:23.760957    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-gxtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:48:23.761135 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:23.761024    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 31 00:48:24.760725 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:24.760686    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-gxtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:48:24.761178 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:24.760747    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 31 00:48:25.594955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:25.594910    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" event={"ID":"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d","Type":"ContainerStarted","Data":"1a1b00915d83b4d80b90c2601a0b3bd4eaa8de8a55246174460f9ec5277356d2"}
Oct 31 00:48:25.722374 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:25.717222    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" podStartSLOduration=2.359123232 podStartE2EDuration="4.717204426s" totalImagesPullingTime="2.358081194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:25.71394002 +0000 UTC m=+1114.268827459" watchObservedRunningTime="2026-10-31 00:48:25.717204426 +0000 UTC m=+1114.272091864"
Oct 31 00:48:25.765376 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:25.764772    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-gxtp4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:48:25.765376 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:25.764833    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 31 00:48:26.094525 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:26.094493    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:26.764218 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:26.764191    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:26.764478 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:26.764451    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:26.765046 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:26.765029    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:48:32.094433 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:32.094397    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:32.106724 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:32.106704    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:32.638963 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:32.638934    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:48:48.551964 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:48:48.551925    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:48:54.590193 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:48:54.590166    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:49:05.573707 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:05.573676    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"]
Oct 31 00:49:05.583334 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:49:05.583286    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="1a1b00915d83b4d80b90c2601a0b3bd4eaa8de8a55246174460f9ec5277356d2" cmd=["/bin/sleep","15"]
Oct 31 00:49:05.583511 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:05.583371    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:49:05.583511 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:49:05.583392    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" podUID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" containerName="main" containerID="cri-o://1a1b00915d83b4d80b90c2601a0b3bd4eaa8de8a55246174460f9ec5277356d2"
Oct 31 00:49:05.583511 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:05.583445    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" podUID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" containerName="main" containerID="cri-o://1a1b00915d83b4d80b90c2601a0b3bd4eaa8de8a55246174460f9ec5277356d2" gracePeriod=60
Oct 31 00:49:05.584501 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:05.584477    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"]
Oct 31 00:49:05.627887 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:05.627849    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" podUID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 31 00:49:05.790109 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:05.790081    4888 generic.go:356] "Generic (PLEG): container finished" podID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" containerID="1a1b00915d83b4d80b90c2601a0b3bd4eaa8de8a55246174460f9ec5277356d2" exitCode=0
Oct 31 00:49:05.790276 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:05.790165    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" event={"ID":"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d","Type":"ContainerDied","Data":"1a1b00915d83b4d80b90c2601a0b3bd4eaa8de8a55246174460f9ec5277356d2"}
Oct 31 00:49:06.329222 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.329202    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:49:06.436762 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.436736    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-kube-api-access-8wllk\" (UniqueName: \"kubernetes.io/projected/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-kube-api-access-8wllk\") pod \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") "
Oct 31 00:49:06.436900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.436779    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-dshm\") pod \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") "
Oct 31 00:49:06.436900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.436808    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tls-certs\") pod \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") "
Oct 31 00:49:06.436900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.436879    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-home\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-home\") pod \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") "
Oct 31 00:49:06.436900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.436896    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-model-cache\") pod \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") "
Oct 31 00:49:06.437110 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.436926    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tmp-dir\") pod \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\" (UID: \"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d\") "
Oct 31 00:49:06.437181 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.437154    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-home" pod "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" (UID: "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:06.437241 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.437197    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-model-cache" pod "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" (UID: "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:06.437241 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.437229    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tmp-dir" pod "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" (UID: "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:06.439123 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.439099    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tls-certs" pod "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" (UID: "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:06.439123 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.439109    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-kube-api-access-8wllk" pod "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" (UID: "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d"). InnerVolumeSpecName "kube-api-access-8wllk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:06.439252 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.439107    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-dshm" pod "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" (UID: "e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:06.537872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.537810    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:06.537872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.537832    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:06.537872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.537861    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:06.537872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.537871    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wllk\" (UniqueName: \"kubernetes.io/projected/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-kube-api-access-8wllk\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:06.538076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.537880    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:06.538076 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.537888    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:06.796223 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.796138    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn" event={"ID":"e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d","Type":"ContainerDied","Data":"e98e757112cbc825c009692871d57604d8caf62fe6161f6b03213e17dc9169c2"}
Oct 31 00:49:06.796223 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.796204    4888 scope.go:122] "RemoveContainer" containerID="1a1b00915d83b4d80b90c2601a0b3bd4eaa8de8a55246174460f9ec5277356d2"
Oct 31 00:49:06.796717 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.796156    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"
Oct 31 00:49:06.959177 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:06.959146    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"]
Oct 31 00:49:07.005748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:07.005725    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-f6ghn"]
Oct 31 00:49:07.558739 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:07.558705    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" path="/var/lib/kubelet/pods/e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d/volumes"
Oct 31 00:49:14.589438 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:14.589407    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:49:18.552229 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:49:18.552192    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:49:20.604066 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:20.603999    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main" containerID="cri-o://e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14" gracePeriod=45
Oct 31 00:49:24.589578 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:24.589541    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:49:34.589969 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:34.589934    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:49:34.590400 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:34.590042    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:49:44.590476 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:44.590441    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:49:48.554296 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:49:48.554256    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:49:50.849031 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:50.849006    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:49:50.905456 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:50.905401    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1040ba98-a930-44d1-986c-bbba87fbbb5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1040ba98-a930-44d1-986c-bbba87fbbb5e-tls-certs\") pod \"1040ba98-a930-44d1-986c-bbba87fbbb5e\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") "
Oct 31 00:49:50.905550 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:50.905480    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1040ba98-a930-44d1-986c-bbba87fbbb5e-kube-api-access-jw5zn\" (UniqueName: \"kubernetes.io/projected/1040ba98-a930-44d1-986c-bbba87fbbb5e-kube-api-access-jw5zn\") pod \"1040ba98-a930-44d1-986c-bbba87fbbb5e\" (UID: \"1040ba98-a930-44d1-986c-bbba87fbbb5e\") "
Oct 31 00:49:50.907603 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:50.907582    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1040ba98-a930-44d1-986c-bbba87fbbb5e-tls-certs" pod "1040ba98-a930-44d1-986c-bbba87fbbb5e" (UID: "1040ba98-a930-44d1-986c-bbba87fbbb5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:50.907654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:50.907617    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1040ba98-a930-44d1-986c-bbba87fbbb5e-kube-api-access-jw5zn" pod "1040ba98-a930-44d1-986c-bbba87fbbb5e" (UID: "1040ba98-a930-44d1-986c-bbba87fbbb5e"). InnerVolumeSpecName "kube-api-access-jw5zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:51.006723 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.006695    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw5zn\" (UniqueName: \"kubernetes.io/projected/1040ba98-a930-44d1-986c-bbba87fbbb5e-kube-api-access-jw5zn\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:51.006723 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.006719    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1040ba98-a930-44d1-986c-bbba87fbbb5e-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:49:51.010840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.010801    4888 generic.go:356] "Generic (PLEG): container finished" podID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerID="e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14" exitCode=0
Oct 31 00:49:51.010957 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.010857    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"
Oct 31 00:49:51.010957 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.010890    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" event={"ID":"1040ba98-a930-44d1-986c-bbba87fbbb5e","Type":"ContainerDied","Data":"e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14"}
Oct 31 00:49:51.010957 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.010947    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w" event={"ID":"1040ba98-a930-44d1-986c-bbba87fbbb5e","Type":"ContainerDied","Data":"9ce0e8d6dcc66f7573f7d2fb4e9c302719b32c1484f70581561787f1886c0c90"}
Oct 31 00:49:51.011066 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.010969    4888 scope.go:122] "RemoveContainer" containerID="e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14"
Oct 31 00:49:51.022446 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.022253    4888 scope.go:122] "RemoveContainer" containerID="e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14"
Oct 31 00:49:51.022987 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:49:51.022958    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14\": container with ID starting with e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14 not found: ID does not exist" containerID="e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14"
Oct 31 00:49:51.023098 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.022994    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14"} err="failed to get container status \"e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14\": rpc error: code = NotFound desc = could not find container \"e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14\": container with ID starting with e61df3b054093985e68cd46ba7d116df5379821fe6f10fa78207c3e005999e14 not found: ID does not exist"
Oct 31 00:49:51.170608 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.170528    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"]
Oct 31 00:49:51.178806 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.178780    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-77fbbb774b-7gl7w"]
Oct 31 00:49:51.553728 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.553696    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" path="/var/lib/kubelet/pods/1040ba98-a930-44d1-986c-bbba87fbbb5e/volumes"
Oct 31 00:49:51.675574 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.675552    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:49:51.678215 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:49:51.678194    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:50:03.722258 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.722228    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"]
Oct 31 00:50:03.722660 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.722585    4888 cpu_manager.go:414] "containerMap: removing container" podUID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" containerName="main"
Oct 31 00:50:03.722660 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.722597    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" containerName="main"
Oct 31 00:50:03.722660 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.722608    4888 cpu_manager.go:414] "containerMap: removing container" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main"
Oct 31 00:50:03.722660 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.722612    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main"
Oct 31 00:50:03.722660 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.722660    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9f4515c-7e9f-4f69-a0c2-39bca2e5b10d" containerName="main"
Oct 31 00:50:03.722822 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.722667    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="1040ba98-a930-44d1-986c-bbba87fbbb5e" containerName="main"
Oct 31 00:50:03.725437 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.725418    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.790256 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.790231    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Oct 31 00:50:03.810663 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.810637    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.810795 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.810682    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.810843 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.810789    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk52j\" (UniqueName: \"kubernetes.io/projected/b84d8ad1-4866-4824-925d-4c9dabad05ca-kube-api-access-lk52j\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.810882 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.810843    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-home\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.810882 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.810874    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84d8ad1-4866-4824-925d-4c9dabad05ca-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.810948 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.810911    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.822862 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.822837    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Oct 31 00:50:03.822862 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.822837    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Oct 31 00:50:03.823022 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.822885    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-qwg99\""
Oct 31 00:50:03.825077 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.825060    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"]
Oct 31 00:50:03.911821 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.911785    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk52j\" (UniqueName: \"kubernetes.io/projected/b84d8ad1-4866-4824-925d-4c9dabad05ca-kube-api-access-lk52j\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.911821 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.911826    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-home\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.912045 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.911871    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84d8ad1-4866-4824-925d-4c9dabad05ca-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.912045 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.911912    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.912045 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.911952    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.912045 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.911997    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.912321 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.912291    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-home\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.912321 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.912305    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-tmp-dir\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.912523 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.912342    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-model-cache\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.914325 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.914299    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-dshm\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.914624 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.914604    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84d8ad1-4866-4824-925d-4c9dabad05ca-tls-certs\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:03.971578 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:03.971545    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk52j\" (UniqueName: \"kubernetes.io/projected/b84d8ad1-4866-4824-925d-4c9dabad05ca-kube-api-access-lk52j\") pod \"fc-concurrency-test-kserve-5485c7876d-qhpdg\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:04.037878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.037792    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:04.246501 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.246472    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"]
Oct 31 00:50:04.250221 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.250183    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.302613 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.302542    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-4rft6\""
Oct 31 00:50:04.311391 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.311342    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"]
Oct 31 00:50:04.315496 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.315470    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832074d6-a9db-42cc-91de-fa69f42bfde8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.315654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.315624    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc6z5\" (UniqueName: \"kubernetes.io/projected/832074d6-a9db-42cc-91de-fa69f42bfde8-kube-api-access-qc6z5\") pod \"fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.416507 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.416473    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832074d6-a9db-42cc-91de-fa69f42bfde8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.416681 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.416543    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc6z5\" (UniqueName: \"kubernetes.io/projected/832074d6-a9db-42cc-91de-fa69f42bfde8-kube-api-access-qc6z5\") pod \"fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.419047 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.419025    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832074d6-a9db-42cc-91de-fa69f42bfde8-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.423708 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.423688    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"]
Oct 31 00:50:04.425822 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:50:04.425801    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84d8ad1_4866_4824_925d_4c9dabad05ca.slice/crio-d4f442b17990f8f573fd99a8d00e21ab4b0b4650b2d504a7e8c98d427ba469b9 WatchSource:0}: Error finding container d4f442b17990f8f573fd99a8d00e21ab4b0b4650b2d504a7e8c98d427ba469b9: Status 404 returned error can't find the container with id d4f442b17990f8f573fd99a8d00e21ab4b0b4650b2d504a7e8c98d427ba469b9
Oct 31 00:50:04.476833 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.476805    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc6z5\" (UniqueName: \"kubernetes.io/projected/832074d6-a9db-42cc-91de-fa69f42bfde8-kube-api-access-qc6z5\") pod \"fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.566380 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.566283    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:04.934486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:04.934408    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"]
Oct 31 00:50:04.937183 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:50:04.937152    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832074d6_a9db_42cc_91de_fa69f42bfde8.slice/crio-99e3782e80cb202f010ea1aafcefb73022668668c77d45a7ab0b5a90d7791433 WatchSource:0}: Error finding container 99e3782e80cb202f010ea1aafcefb73022668668c77d45a7ab0b5a90d7791433: Status 404 returned error can't find the container with id 99e3782e80cb202f010ea1aafcefb73022668668c77d45a7ab0b5a90d7791433
Oct 31 00:50:05.077284 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:05.077247    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" event={"ID":"832074d6-a9db-42cc-91de-fa69f42bfde8","Type":"ContainerStarted","Data":"f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01"}
Oct 31 00:50:05.077502 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:05.077304    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" event={"ID":"832074d6-a9db-42cc-91de-fa69f42bfde8","Type":"ContainerStarted","Data":"99e3782e80cb202f010ea1aafcefb73022668668c77d45a7ab0b5a90d7791433"}
Oct 31 00:50:05.077502 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:05.077330    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:05.078974 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:05.078950    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" event={"ID":"b84d8ad1-4866-4824-925d-4c9dabad05ca","Type":"ContainerStarted","Data":"d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075"}
Oct 31 00:50:05.079082 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:05.078990    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" event={"ID":"b84d8ad1-4866-4824-925d-4c9dabad05ca","Type":"ContainerStarted","Data":"d4f442b17990f8f573fd99a8d00e21ab4b0b4650b2d504a7e8c98d427ba469b9"}
Oct 31 00:50:05.190138 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:05.190046    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" podStartSLOduration=1.190031413 podStartE2EDuration="1.190031413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:05.188860871 +0000 UTC m=+1213.743748331" watchObservedRunningTime="2026-10-31 00:50:05.190031413 +0000 UTC m=+1213.744918851"
Oct 31 00:50:05.307681 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:05.307622    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" podStartSLOduration=2.307602103 podStartE2EDuration="2.307602103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:05.305726723 +0000 UTC m=+1213.860614162" watchObservedRunningTime="2026-10-31 00:50:05.307602103 +0000 UTC m=+1213.862489543"
Oct 31 00:50:06.038817 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:06.038768    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:14.038105 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:14.038065    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:14.050527 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:14.050503    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:14.133975 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:14.133951    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:18.551806 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:50:18.551759    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:50:36.084850 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:36.084824    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:50:46.728268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:46.728230    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"]
Oct 31 00:50:46.731702 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:46.731675    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"]
Oct 31 00:50:46.743422 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:50:46.743385    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075" cmd=["/bin/sleep","15"]
Oct 31 00:50:46.743541 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:46.743439    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:50:46.743541 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:50:46.743455    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" podUID="b84d8ad1-4866-4824-925d-4c9dabad05ca" containerName="main" containerID="cri-o://d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075"
Oct 31 00:50:46.743541 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:46.743502    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" podUID="b84d8ad1-4866-4824-925d-4c9dabad05ca" containerName="main" containerID="cri-o://d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075" gracePeriod=60
Oct 31 00:50:47.088937 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.088912    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:47.197695 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.197665    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84d8ad1-4866-4824-925d-4c9dabad05ca-kube-api-access-lk52j\" (UniqueName: \"kubernetes.io/projected/b84d8ad1-4866-4824-925d-4c9dabad05ca-kube-api-access-lk52j\") pod \"b84d8ad1-4866-4824-925d-4c9dabad05ca\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") "
Oct 31 00:50:47.197867 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.197719    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-home\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-home\") pod \"b84d8ad1-4866-4824-925d-4c9dabad05ca\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") "
Oct 31 00:50:47.197867 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.197752    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-dshm\") pod \"b84d8ad1-4866-4824-925d-4c9dabad05ca\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") "
Oct 31 00:50:47.197867 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.197848    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-model-cache\") pod \"b84d8ad1-4866-4824-925d-4c9dabad05ca\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") "
Oct 31 00:50:47.198010 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.197875    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b84d8ad1-4866-4824-925d-4c9dabad05ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/b84d8ad1-4866-4824-925d-4c9dabad05ca-tls-certs\") pod \"b84d8ad1-4866-4824-925d-4c9dabad05ca\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") "
Oct 31 00:50:47.198010 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.197925    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-tmp-dir\") pod \"b84d8ad1-4866-4824-925d-4c9dabad05ca\" (UID: \"b84d8ad1-4866-4824-925d-4c9dabad05ca\") "
Oct 31 00:50:47.198113 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.198022    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-home" pod "b84d8ad1-4866-4824-925d-4c9dabad05ca" (UID: "b84d8ad1-4866-4824-925d-4c9dabad05ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:47.198169 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.198101    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-model-cache" pod "b84d8ad1-4866-4824-925d-4c9dabad05ca" (UID: "b84d8ad1-4866-4824-925d-4c9dabad05ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:47.198243 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.198222    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-tmp-dir" pod "b84d8ad1-4866-4824-925d-4c9dabad05ca" (UID: "b84d8ad1-4866-4824-925d-4c9dabad05ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:47.198299 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.198283    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:47.198340 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.198304    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:47.200051 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.200030    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b84d8ad1-4866-4824-925d-4c9dabad05ca-tls-certs" pod "b84d8ad1-4866-4824-925d-4c9dabad05ca" (UID: "b84d8ad1-4866-4824-925d-4c9dabad05ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:50:47.200128 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.200108    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-dshm" pod "b84d8ad1-4866-4824-925d-4c9dabad05ca" (UID: "b84d8ad1-4866-4824-925d-4c9dabad05ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:50:47.200128 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.200118    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84d8ad1-4866-4824-925d-4c9dabad05ca-kube-api-access-lk52j" pod "b84d8ad1-4866-4824-925d-4c9dabad05ca" (UID: "b84d8ad1-4866-4824-925d-4c9dabad05ca"). InnerVolumeSpecName "kube-api-access-lk52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:50:47.290042 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.289953    4888 generic.go:356] "Generic (PLEG): container finished" podID="b84d8ad1-4866-4824-925d-4c9dabad05ca" containerID="d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075" exitCode=0
Oct 31 00:50:47.290042 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.289999    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"
Oct 31 00:50:47.290257 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.290040    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" event={"ID":"b84d8ad1-4866-4824-925d-4c9dabad05ca","Type":"ContainerDied","Data":"d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075"}
Oct 31 00:50:47.290257 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.290085    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg" event={"ID":"b84d8ad1-4866-4824-925d-4c9dabad05ca","Type":"ContainerDied","Data":"d4f442b17990f8f573fd99a8d00e21ab4b0b4650b2d504a7e8c98d427ba469b9"}
Oct 31 00:50:47.290257 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.290104    4888 scope.go:122] "RemoveContainer" containerID="d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075"
Oct 31 00:50:47.299451 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.299428    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:47.299451 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.299450    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b84d8ad1-4866-4824-925d-4c9dabad05ca-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:47.299586 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.299459    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b84d8ad1-4866-4824-925d-4c9dabad05ca-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:47.299586 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.299456    4888 scope.go:122] "RemoveContainer" containerID="d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075"
Oct 31 00:50:47.299586 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.299467    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk52j\" (UniqueName: \"kubernetes.io/projected/b84d8ad1-4866-4824-925d-4c9dabad05ca-kube-api-access-lk52j\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:50:47.299761 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:50:47.299732    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075\": container with ID starting with d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075 not found: ID does not exist" containerID="d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075"
Oct 31 00:50:47.299798 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.299768    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075"} err="failed to get container status \"d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075\": rpc error: code = NotFound desc = could not find container \"d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075\": container with ID starting with d0286a1dfd768b7b86e727277881a49604289347ef2a0a5651ab4fc0c63a9075 not found: ID does not exist"
Oct 31 00:50:47.452126 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.450288    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"]
Oct 31 00:50:47.455519 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.455494    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-5485c7876d-qhpdg"]
Oct 31 00:50:47.554596 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:47.554524    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84d8ad1-4866-4824-925d-4c9dabad05ca" path="/var/lib/kubelet/pods/b84d8ad1-4866-4824-925d-4c9dabad05ca/volumes"
Oct 31 00:50:48.551851 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:50:48.551816    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:50:56.084869 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:50:56.084824    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:51:01.750186 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:01.750139    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main" containerID="cri-o://f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01" gracePeriod=45
Oct 31 00:51:06.085192 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:06.085159    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:51:16.084837 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:16.084800    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:51:16.085287 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:16.084908    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:51:18.552565 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:18.552528    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:51:26.084904 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:26.084815    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:51:31.998014 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:31.997991    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:51:32.079404 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.079309    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832074d6-a9db-42cc-91de-fa69f42bfde8-kube-api-access-qc6z5\" (UniqueName: \"kubernetes.io/projected/832074d6-a9db-42cc-91de-fa69f42bfde8-kube-api-access-qc6z5\") pod \"832074d6-a9db-42cc-91de-fa69f42bfde8\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") "
Oct 31 00:51:32.079404 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.079382    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/832074d6-a9db-42cc-91de-fa69f42bfde8-tls-certs\" (UniqueName: \"kubernetes.io/secret/832074d6-a9db-42cc-91de-fa69f42bfde8-tls-certs\") pod \"832074d6-a9db-42cc-91de-fa69f42bfde8\" (UID: \"832074d6-a9db-42cc-91de-fa69f42bfde8\") "
Oct 31 00:51:32.081540 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.081510    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832074d6-a9db-42cc-91de-fa69f42bfde8-tls-certs" pod "832074d6-a9db-42cc-91de-fa69f42bfde8" (UID: "832074d6-a9db-42cc-91de-fa69f42bfde8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:51:32.081637 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.081594    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832074d6-a9db-42cc-91de-fa69f42bfde8-kube-api-access-qc6z5" pod "832074d6-a9db-42cc-91de-fa69f42bfde8" (UID: "832074d6-a9db-42cc-91de-fa69f42bfde8"). InnerVolumeSpecName "kube-api-access-qc6z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:32.180209 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.180182    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/832074d6-a9db-42cc-91de-fa69f42bfde8-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:32.180209 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.180205    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc6z5\" (UniqueName: \"kubernetes.io/projected/832074d6-a9db-42cc-91de-fa69f42bfde8-kube-api-access-qc6z5\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:32.501099 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.501069    4888 generic.go:356] "Generic (PLEG): container finished" podID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerID="f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01" exitCode=0
Oct 31 00:51:32.501262 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.501115    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"
Oct 31 00:51:32.501262 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.501162    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" event={"ID":"832074d6-a9db-42cc-91de-fa69f42bfde8","Type":"ContainerDied","Data":"f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01"}
Oct 31 00:51:32.501262 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.501234    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb" event={"ID":"832074d6-a9db-42cc-91de-fa69f42bfde8","Type":"ContainerDied","Data":"99e3782e80cb202f010ea1aafcefb73022668668c77d45a7ab0b5a90d7791433"}
Oct 31 00:51:32.501262 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.501255    4888 scope.go:122] "RemoveContainer" containerID="f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01"
Oct 31 00:51:32.510665 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.510649    4888 scope.go:122] "RemoveContainer" containerID="f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01"
Oct 31 00:51:32.510920 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:32.510900    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01\": container with ID starting with f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01 not found: ID does not exist" containerID="f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01"
Oct 31 00:51:32.510986 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.510933    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01"} err="failed to get container status \"f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01\": rpc error: code = NotFound desc = could not find container \"f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01\": container with ID starting with f16b30a67a97e456134615d9a52c389d1a7c06a1f42348b2770324633801be01 not found: ID does not exist"
Oct 31 00:51:32.659703 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.659677    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"]
Oct 31 00:51:32.706635 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:32.706603    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-7db7c65fd7-pnqnb"]
Oct 31 00:51:33.553538 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:33.553508    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" path="/var/lib/kubelet/pods/832074d6-a9db-42cc-91de-fa69f42bfde8/volumes"
Oct 31 00:51:36.840873 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.840845    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"]
Oct 31 00:51:36.841263 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.841166    4888 cpu_manager.go:414] "containerMap: removing container" podUID="b84d8ad1-4866-4824-925d-4c9dabad05ca" containerName="main"
Oct 31 00:51:36.841263 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.841177    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84d8ad1-4866-4824-925d-4c9dabad05ca" containerName="main"
Oct 31 00:51:36.841263 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.841199    4888 cpu_manager.go:414] "containerMap: removing container" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main"
Oct 31 00:51:36.841263 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.841204    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main"
Oct 31 00:51:36.841263 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.841261    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84d8ad1-4866-4824-925d-4c9dabad05ca" containerName="main"
Oct 31 00:51:36.841515 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.841271    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="832074d6-a9db-42cc-91de-fa69f42bfde8" containerName="main"
Oct 31 00:51:36.845484 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.845465    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916089 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916065    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-data\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916089 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916093    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-token\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916300 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916114    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916300 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916130    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-credential-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916300 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916239    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhkl6\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-kube-api-access-zhkl6\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916300 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916277    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-certs\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916477 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916378    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-podinfo\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916477 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916399    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486c40bf-a32e-437c-981e-5eeb29a888f4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.916477 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.916419    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-envoy\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:36.924914 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.924896    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-szvkq\""
Oct 31 00:51:36.925115 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.925098    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Oct 31 00:51:36.925159 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.925148    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Oct 31 00:51:36.925159 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.925151    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:51:36.944960 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:36.944941    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"]
Oct 31 00:51:37.017812 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.017780    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhkl6\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-kube-api-access-zhkl6\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.017991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.017824    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-certs\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.017991 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.017869    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-podinfo\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.018101 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018020    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486c40bf-a32e-437c-981e-5eeb29a888f4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.018101 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018056    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-envoy\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.018209 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018115    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-data\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.018209 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018138    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-token\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.018209 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018171    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.018886 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018763    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-certs\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.018886 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018816    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.019063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018917    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-data\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.019063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.018193    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-credential-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.021032 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.019170    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486c40bf-a32e-437c-981e-5eeb29a888f4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.021032 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.019455    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-credential-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.025012 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.021889    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-podinfo\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.025809 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.025791    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-envoy\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.077896 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.077860    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-token\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.078007 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.077967    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhkl6\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-kube-api-access-zhkl6\") pod \"router-gateway-1-openshift-default-dcc9f996f-k8m6v\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.157759 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.157684    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:37.573109 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.573086    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"]
Oct 31 00:51:37.575154 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:51:37.575124    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486c40bf_a32e_437c_981e_5eeb29a888f4.slice/crio-78dd9e78d8ef69cad21ef5955fa78f241597c421af7d509394a5eceec5446601 WatchSource:0}: Error finding container 78dd9e78d8ef69cad21ef5955fa78f241597c421af7d509394a5eceec5446601: Status 404 returned error can't find the container with id 78dd9e78d8ef69cad21ef5955fa78f241597c421af7d509394a5eceec5446601
Oct 31 00:51:37.577056 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.577026    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:51:37.577128 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.577087    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:51:37.577128 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:37.577118    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 00:51:38.531723 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:38.531692    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v" event={"ID":"486c40bf-a32e-437c-981e-5eeb29a888f4","Type":"ContainerStarted","Data":"37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3"}
Oct 31 00:51:38.532110 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:38.531740    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v" event={"ID":"486c40bf-a32e-437c-981e-5eeb29a888f4","Type":"ContainerStarted","Data":"78dd9e78d8ef69cad21ef5955fa78f241597c421af7d509394a5eceec5446601"}
Oct 31 00:51:38.695747 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:38.695694    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v" podStartSLOduration=2.695680424 podStartE2EDuration="2.695680424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:38.693782715 +0000 UTC m=+1307.248670164" watchObservedRunningTime="2026-10-31 00:51:38.695680424 +0000 UTC m=+1307.250567924"
Oct 31 00:51:39.158271 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:39.158239    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:39.163014 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:39.162989    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:39.537269 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:39.537239    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:39.538232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:39.538211    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:45.323573 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.323542    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"]
Oct 31 00:51:45.327341 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.327324    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.391492 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.391465    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grdqz\" (UniqueName: \"kubernetes.io/projected/dacdf247-cb48-4791-877c-28205775b50c-kube-api-access-grdqz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.391636 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.391500    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.391686 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.391666    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.391726 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.391713    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.391769 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.391760    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.391806 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.391779    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.424955 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.424927    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-q4479\""
Oct 31 00:51:45.425080 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.424986    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Oct 31 00:51:45.427006 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.426988    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"]
Oct 31 00:51:45.492699 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.492665    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.492911 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.492703    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.492911 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.492746    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grdqz\" (UniqueName: \"kubernetes.io/projected/dacdf247-cb48-4791-877c-28205775b50c-kube-api-access-grdqz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.492911 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.492773    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.492911 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.492836    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.492911 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.492863    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.493577 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.493046    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.493577 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.493132    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.493577 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.493462    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.495531 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.495500    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.495815 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.495798    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.553838 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.553809    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grdqz\" (UniqueName: \"kubernetes.io/projected/dacdf247-cb48-4791-877c-28205775b50c-kube-api-access-grdqz\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:45.639599 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:45.639517    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:46.013033 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:46.013008    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"]
Oct 31 00:51:46.015246 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:51:46.015214    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacdf247_cb48_4791_877c_28205775b50c.slice/crio-acbe7aec9e1fa7bce136e077304aa8d1752f9cb4dd6d534eed1e1018dd019864 WatchSource:0}: Error finding container acbe7aec9e1fa7bce136e077304aa8d1752f9cb4dd6d534eed1e1018dd019864: Status 404 returned error can't find the container with id acbe7aec9e1fa7bce136e077304aa8d1752f9cb4dd6d534eed1e1018dd019864
Oct 31 00:51:46.571727 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:46.571689    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" event={"ID":"dacdf247-cb48-4791-877c-28205775b50c","Type":"ContainerStarted","Data":"c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431"}
Oct 31 00:51:46.572123 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:46.571753    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" event={"ID":"dacdf247-cb48-4791-877c-28205775b50c","Type":"ContainerStarted","Data":"acbe7aec9e1fa7bce136e077304aa8d1752f9cb4dd6d534eed1e1018dd019864"}
Oct 31 00:51:46.640271 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:46.640240    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:46.685389 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:46.685318    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" podStartSLOduration=1.68530717 podStartE2EDuration="1.68530717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:46.685149709 +0000 UTC m=+1315.240037168" watchObservedRunningTime="2026-10-31 00:51:46.68530717 +0000 UTC m=+1315.240194608"
Oct 31 00:51:48.148415 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:48.148383    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"]
Oct 31 00:51:48.551995 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:48.551959    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:51:48.625017 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:48.624985    4888 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 31 00:51:48.625200 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:48.625058    4888 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs podName:dacdf247-cb48-4791-877c-28205775b50c nodeName:}" failed. No retries permitted until 2026-10-31 00:51:49.125043776 +0000 UTC m=+1317.679931192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" (UID: "dacdf247-cb48-4791-877c-28205775b50c") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 31 00:51:49.131152 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:49.131114    4888 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-49b5edd9/gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs: secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 31 00:51:49.131383 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:49.131192    4888 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs podName:dacdf247-cb48-4791-877c-28205775b50c nodeName:}" failed. No retries permitted until 2026-10-31 00:51:50.131175015 +0000 UTC m=+1318.686062431 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs") pod "gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" (UID: "dacdf247-cb48-4791-877c-28205775b50c") : secret "gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs" not found
Oct 31 00:51:49.595446 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:49.595411    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431" cmd=["/bin/sleep","15"]
Oct 31 00:51:49.595837 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:49.595457    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:51:49.595837 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:49.595469    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" podUID="dacdf247-cb48-4791-877c-28205775b50c" containerName="main" containerID="cri-o://c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431"
Oct 31 00:51:49.595837 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:49.595516    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" podUID="dacdf247-cb48-4791-877c-28205775b50c" containerName="main" containerID="cri-o://c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431" gracePeriod=60
Oct 31 00:51:49.940030 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:49.940005    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:50.038513 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038483    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-home\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-home\") pod \"dacdf247-cb48-4791-877c-28205775b50c\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") "
Oct 31 00:51:50.038705 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038547    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs\" (UniqueName: \"kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs\") pod \"dacdf247-cb48-4791-877c-28205775b50c\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") "
Oct 31 00:51:50.038705 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038605    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-dshm\") pod \"dacdf247-cb48-4791-877c-28205775b50c\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") "
Oct 31 00:51:50.038705 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038649    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dacdf247-cb48-4791-877c-28205775b50c-kube-api-access-grdqz\" (UniqueName: \"kubernetes.io/projected/dacdf247-cb48-4791-877c-28205775b50c-kube-api-access-grdqz\") pod \"dacdf247-cb48-4791-877c-28205775b50c\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") "
Oct 31 00:51:50.038705 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038685    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-model-cache\") pod \"dacdf247-cb48-4791-877c-28205775b50c\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") "
Oct 31 00:51:50.038930 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038711    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-tmp-dir\") pod \"dacdf247-cb48-4791-877c-28205775b50c\" (UID: \"dacdf247-cb48-4791-877c-28205775b50c\") "
Oct 31 00:51:50.038930 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038757    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-home" pod "dacdf247-cb48-4791-877c-28205775b50c" (UID: "dacdf247-cb48-4791-877c-28205775b50c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:50.039026 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038955    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-model-cache" pod "dacdf247-cb48-4791-877c-28205775b50c" (UID: "dacdf247-cb48-4791-877c-28205775b50c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:50.039026 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.038993    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-tmp-dir" pod "dacdf247-cb48-4791-877c-28205775b50c" (UID: "dacdf247-cb48-4791-877c-28205775b50c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:50.039094 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.039038    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:50.039094 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.039056    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:50.040987 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.040962    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-dshm" pod "dacdf247-cb48-4791-877c-28205775b50c" (UID: "dacdf247-cb48-4791-877c-28205775b50c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:50.041086 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.041028    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dacdf247-cb48-4791-877c-28205775b50c-kube-api-access-grdqz" pod "dacdf247-cb48-4791-877c-28205775b50c" (UID: "dacdf247-cb48-4791-877c-28205775b50c"). InnerVolumeSpecName "kube-api-access-grdqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:50.041086 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.041080    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs" pod "dacdf247-cb48-4791-877c-28205775b50c" (UID: "dacdf247-cb48-4791-877c-28205775b50c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:51:50.140327 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.140250    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:50.140327 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.140275    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dacdf247-cb48-4791-877c-28205775b50c-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:50.140327 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.140283    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dacdf247-cb48-4791-877c-28205775b50c-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:50.140327 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.140294    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grdqz\" (UniqueName: \"kubernetes.io/projected/dacdf247-cb48-4791-877c-28205775b50c-kube-api-access-grdqz\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:50.592620 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.592590    4888 generic.go:356] "Generic (PLEG): container finished" podID="dacdf247-cb48-4791-877c-28205775b50c" containerID="c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431" exitCode=0
Oct 31 00:51:50.592821 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.592636    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"
Oct 31 00:51:50.592821 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.592637    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" event={"ID":"dacdf247-cb48-4791-877c-28205775b50c","Type":"ContainerDied","Data":"c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431"}
Oct 31 00:51:50.592821 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.592778    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb" event={"ID":"dacdf247-cb48-4791-877c-28205775b50c","Type":"ContainerDied","Data":"acbe7aec9e1fa7bce136e077304aa8d1752f9cb4dd6d534eed1e1018dd019864"}
Oct 31 00:51:50.592821 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.592816    4888 scope.go:122] "RemoveContainer" containerID="c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431"
Oct 31 00:51:50.602244 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.602046    4888 scope.go:122] "RemoveContainer" containerID="c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431"
Oct 31 00:51:50.602507 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:50.602283    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431\": container with ID starting with c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431 not found: ID does not exist" containerID="c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431"
Oct 31 00:51:50.602507 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.602308    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431"} err="failed to get container status \"c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431\": rpc error: code = NotFound desc = could not find container \"c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431\": container with ID starting with c1aea5ae038971bd77b6155594679620f704297fb8b2dd56f4eaeca1bec79431 not found: ID does not exist"
Oct 31 00:51:50.754595 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.754561    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"]
Oct 31 00:51:50.802080 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:50.802047    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-87cb66947-xjtpb"]
Oct 31 00:51:51.553951 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:51.553918    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dacdf247-cb48-4791-877c-28205775b50c" path="/var/lib/kubelet/pods/dacdf247-cb48-4791-877c-28205775b50c/volumes"
Oct 31 00:51:52.548859 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:52.548828    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"]
Oct 31 00:51:52.549314 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:52.549019    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v" podUID="486c40bf-a32e-437c-981e-5eeb29a888f4" containerName="istio-proxy" containerID="cri-o://37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3" gracePeriod=30
Oct 31 00:51:54.539768 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:54.539730    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v" podUID="486c40bf-a32e-437c-981e-5eeb29a888f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:51:56.124478 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.124440    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"]
Oct 31 00:51:56.125022 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.124957    4888 cpu_manager.go:414] "containerMap: removing container" podUID="dacdf247-cb48-4791-877c-28205775b50c" containerName="main"
Oct 31 00:51:56.125022 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.124977    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dacdf247-cb48-4791-877c-28205775b50c" containerName="main"
Oct 31 00:51:56.125145 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.125076    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="dacdf247-cb48-4791-877c-28205775b50c" containerName="main"
Oct 31 00:51:56.129308 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.129286    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.193617 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.193588    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43eb961-439a-4815-a4e1-8d6e9349fad8-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.193807 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.193634    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.193807 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.193657    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbnhk\" (UniqueName: \"kubernetes.io/projected/f43eb961-439a-4815-a4e1-8d6e9349fad8-kube-api-access-lbnhk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.193807 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.193788    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.193807 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.193809    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.194036 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.193834    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.226464 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.226438    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-6wtgb\""
Oct 31 00:51:56.226695 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.226632    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Oct 31 00:51:56.226774 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.226727    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Oct 31 00:51:56.228925 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.228901    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"]
Oct 31 00:51:56.280671 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.280644    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Oct 31 00:51:56.294638 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.294610    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.294770 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.294646    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.294843 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.294779    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43eb961-439a-4815-a4e1-8d6e9349fad8-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.294900 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.294867    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.294953 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.294904    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbnhk\" (UniqueName: \"kubernetes.io/projected/f43eb961-439a-4815-a4e1-8d6e9349fad8-kube-api-access-lbnhk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.295007 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.294949    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.295056 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.295005    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.295197 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.295172    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.295411 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.295329    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.297093 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.297075    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.297235 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.297221    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43eb961-439a-4815-a4e1-8d6e9349fad8-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.355776 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.355749    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbnhk\" (UniqueName: \"kubernetes.io/projected/f43eb961-439a-4815-a4e1-8d6e9349fad8-kube-api-access-lbnhk\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.441961 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.441871    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:56.832506 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:56.832480    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"]
Oct 31 00:51:56.834520 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:51:56.834496    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf43eb961_439a_4815_a4e1_8d6e9349fad8.slice/crio-55cd45441c3b2de10ded339333b56792f9fc659c6d231eda4907d2cd949b4562 WatchSource:0}: Error finding container 55cd45441c3b2de10ded339333b56792f9fc659c6d231eda4907d2cd949b4562: Status 404 returned error can't find the container with id 55cd45441c3b2de10ded339333b56792f9fc659c6d231eda4907d2cd949b4562
Oct 31 00:51:57.628593 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.628563    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" event={"ID":"f43eb961-439a-4815-a4e1-8d6e9349fad8","Type":"ContainerStarted","Data":"64c1eafa714ccd07f33e30f0aacacc6eb9c6b2bd859b81b9397a197ed353d8d9"}
Oct 31 00:51:57.628906 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.628609    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" event={"ID":"f43eb961-439a-4815-a4e1-8d6e9349fad8","Type":"ContainerStarted","Data":"55cd45441c3b2de10ded339333b56792f9fc659c6d231eda4907d2cd949b4562"}
Oct 31 00:51:57.743390 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.743316    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" podStartSLOduration=1.743297428 podStartE2EDuration="1.743297428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:57.742792003 +0000 UTC m=+1326.297679449" watchObservedRunningTime="2026-10-31 00:51:57.743297428 +0000 UTC m=+1326.298184865"
Oct 31 00:51:57.786242 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.786219    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:57.908184 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908109    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-credential-socket\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908184 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908155    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-token\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-token\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908448 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908189    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-certs\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908448 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908254    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-data\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908448 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908287    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486c40bf-a32e-437c-981e-5eeb29a888f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486c40bf-a32e-437c-981e-5eeb29a888f4-istiod-ca-cert\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908448 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908322    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-socket\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908448 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908391    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-kube-api-access-zhkl6\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-kube-api-access-zhkl6\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908448 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908421    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-podinfo\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908754 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908453    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-envoy\") pod \"486c40bf-a32e-437c-981e-5eeb29a888f4\" (UID: \"486c40bf-a32e-437c-981e-5eeb29a888f4\") "
Oct 31 00:51:57.908754 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908475    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-credential-socket" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:57.908754 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908561    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-certs" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:57.908754 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908578    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-data" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:57.908754 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908747    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:57.909033 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908755    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-socket" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:57.909033 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908765    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:57.909033 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.908780    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:57.909375 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.909142    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486c40bf-a32e-437c-981e-5eeb29a888f4-istiod-ca-cert" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:51:57.910678 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.910655    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-kube-api-access-zhkl6" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "kube-api-access-zhkl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:57.910831 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.910747    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-podinfo" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:51:57.910831 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.910800    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-token" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:51:57.922722 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:57.922697    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-envoy" pod "486c40bf-a32e-437c-981e-5eeb29a888f4" (UID: "486c40bf-a32e-437c-981e-5eeb29a888f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:51:58.009233 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.009206    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486c40bf-a32e-437c-981e-5eeb29a888f4-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:58.009233 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.009230    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:58.009417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.009240    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhkl6\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-kube-api-access-zhkl6\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:58.009417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.009250    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:58.009417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.009258    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:58.009417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.009267    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486c40bf-a32e-437c-981e-5eeb29a888f4-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:51:58.442203 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.442169    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:51:58.634609 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.634578    4888 generic.go:356] "Generic (PLEG): container finished" podID="486c40bf-a32e-437c-981e-5eeb29a888f4" containerID="37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3" exitCode=0
Oct 31 00:51:58.635054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.634635    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"
Oct 31 00:51:58.635054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.634679    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v" event={"ID":"486c40bf-a32e-437c-981e-5eeb29a888f4","Type":"ContainerDied","Data":"37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3"}
Oct 31 00:51:58.635054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.634733    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v" event={"ID":"486c40bf-a32e-437c-981e-5eeb29a888f4","Type":"ContainerDied","Data":"78dd9e78d8ef69cad21ef5955fa78f241597c421af7d509394a5eceec5446601"}
Oct 31 00:51:58.635054 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.634755    4888 scope.go:122] "RemoveContainer" containerID="37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3"
Oct 31 00:51:58.645011 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.644992    4888 scope.go:122] "RemoveContainer" containerID="37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3"
Oct 31 00:51:58.645285 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:51:58.645267    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3\": container with ID starting with 37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3 not found: ID does not exist" containerID="37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3"
Oct 31 00:51:58.645327 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.645295    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3"} err="failed to get container status \"37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3\": rpc error: code = NotFound desc = could not find container \"37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3\": container with ID starting with 37491536f1880f37dc937d8d0d86dfd9d364ed2daf8038a4144cca1e00e8b5f3 not found: ID does not exist"
Oct 31 00:51:58.840878 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.840845    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"]
Oct 31 00:51:58.887834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:58.887808    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-k8m6v"]
Oct 31 00:51:59.553711 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:51:59.553676    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486c40bf-a32e-437c-981e-5eeb29a888f4" path="/var/lib/kubelet/pods/486c40bf-a32e-437c-981e-5eeb29a888f4/volumes"
Oct 31 00:52:02.438790 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.438755    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"]
Oct 31 00:52:02.447860 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:52:02.447821    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="64c1eafa714ccd07f33e30f0aacacc6eb9c6b2bd859b81b9397a197ed353d8d9" cmd=["/bin/sleep","15"]
Oct 31 00:52:02.447972 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.447872    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:52:02.447972 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:52:02.447884    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" podUID="f43eb961-439a-4815-a4e1-8d6e9349fad8" containerName="main" containerID="cri-o://64c1eafa714ccd07f33e30f0aacacc6eb9c6b2bd859b81b9397a197ed353d8d9"
Oct 31 00:52:02.447972 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.447918    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" podUID="f43eb961-439a-4815-a4e1-8d6e9349fad8" containerName="main" containerID="cri-o://64c1eafa714ccd07f33e30f0aacacc6eb9c6b2bd859b81b9397a197ed353d8d9" gracePeriod=60
Oct 31 00:52:02.659057 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.659028    4888 generic.go:356] "Generic (PLEG): container finished" podID="f43eb961-439a-4815-a4e1-8d6e9349fad8" containerID="64c1eafa714ccd07f33e30f0aacacc6eb9c6b2bd859b81b9397a197ed353d8d9" exitCode=0
Oct 31 00:52:02.659184 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.659098    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" event={"ID":"f43eb961-439a-4815-a4e1-8d6e9349fad8","Type":"ContainerDied","Data":"64c1eafa714ccd07f33e30f0aacacc6eb9c6b2bd859b81b9397a197ed353d8d9"}
Oct 31 00:52:02.708457 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.708432    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:52:02.847436 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.847401    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-model-cache\") pod \"f43eb961-439a-4815-a4e1-8d6e9349fad8\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") "
Oct 31 00:52:02.847601 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.847483    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43eb961-439a-4815-a4e1-8d6e9349fad8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f43eb961-439a-4815-a4e1-8d6e9349fad8-tls-certs\") pod \"f43eb961-439a-4815-a4e1-8d6e9349fad8\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") "
Oct 31 00:52:02.847601 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.847570    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-dshm\") pod \"f43eb961-439a-4815-a4e1-8d6e9349fad8\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") "
Oct 31 00:52:02.847714 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.847600    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-home\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-home\") pod \"f43eb961-439a-4815-a4e1-8d6e9349fad8\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") "
Oct 31 00:52:02.847714 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.847610    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-model-cache" pod "f43eb961-439a-4815-a4e1-8d6e9349fad8" (UID: "f43eb961-439a-4815-a4e1-8d6e9349fad8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:02.847714 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.847686    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-tmp-dir\") pod \"f43eb961-439a-4815-a4e1-8d6e9349fad8\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") "
Oct 31 00:52:02.847825 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.847762    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43eb961-439a-4815-a4e1-8d6e9349fad8-kube-api-access-lbnhk\" (UniqueName: \"kubernetes.io/projected/f43eb961-439a-4815-a4e1-8d6e9349fad8-kube-api-access-lbnhk\") pod \"f43eb961-439a-4815-a4e1-8d6e9349fad8\" (UID: \"f43eb961-439a-4815-a4e1-8d6e9349fad8\") "
Oct 31 00:52:02.848061 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.848032    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:02.848061 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.848032    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-home" pod "f43eb961-439a-4815-a4e1-8d6e9349fad8" (UID: "f43eb961-439a-4815-a4e1-8d6e9349fad8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:02.848382 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.848341    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-tmp-dir" pod "f43eb961-439a-4815-a4e1-8d6e9349fad8" (UID: "f43eb961-439a-4815-a4e1-8d6e9349fad8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:02.851827 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.851804    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43eb961-439a-4815-a4e1-8d6e9349fad8-kube-api-access-lbnhk" pod "f43eb961-439a-4815-a4e1-8d6e9349fad8" (UID: "f43eb961-439a-4815-a4e1-8d6e9349fad8"). InnerVolumeSpecName "kube-api-access-lbnhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:52:02.859486 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.859454    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43eb961-439a-4815-a4e1-8d6e9349fad8-tls-certs" pod "f43eb961-439a-4815-a4e1-8d6e9349fad8" (UID: "f43eb961-439a-4815-a4e1-8d6e9349fad8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:52:02.862501 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.862471    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-dshm" pod "f43eb961-439a-4815-a4e1-8d6e9349fad8" (UID: "f43eb961-439a-4815-a4e1-8d6e9349fad8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:52:02.948454 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.948422    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbnhk\" (UniqueName: \"kubernetes.io/projected/f43eb961-439a-4815-a4e1-8d6e9349fad8-kube-api-access-lbnhk\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:02.948454 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.948453    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f43eb961-439a-4815-a4e1-8d6e9349fad8-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:02.948643 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.948467    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:02.948643 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.948477    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:02.948643 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:02.948488    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f43eb961-439a-4815-a4e1-8d6e9349fad8-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:52:03.671150 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:03.671125    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"
Oct 31 00:52:03.671553 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:03.671158    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v" event={"ID":"f43eb961-439a-4815-a4e1-8d6e9349fad8","Type":"ContainerDied","Data":"55cd45441c3b2de10ded339333b56792f9fc659c6d231eda4907d2cd949b4562"}
Oct 31 00:52:03.671553 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:03.671241    4888 scope.go:122] "RemoveContainer" containerID="64c1eafa714ccd07f33e30f0aacacc6eb9c6b2bd859b81b9397a197ed353d8d9"
Oct 31 00:52:03.829405 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:03.829307    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"]
Oct 31 00:52:03.875926 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:03.875897    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-cf8f45b46-nfv2v"]
Oct 31 00:52:05.553343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.553314    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43eb961-439a-4815-a4e1-8d6e9349fad8" path="/var/lib/kubelet/pods/f43eb961-439a-4815-a4e1-8d6e9349fad8/volumes"
Oct 31 00:52:05.817552 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.817477    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"]
Oct 31 00:52:05.817834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.817817    4888 cpu_manager.go:414] "containerMap: removing container" podUID="f43eb961-439a-4815-a4e1-8d6e9349fad8" containerName="main"
Oct 31 00:52:05.817834 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.817833    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43eb961-439a-4815-a4e1-8d6e9349fad8" containerName="main"
Oct 31 00:52:05.817982 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.817868    4888 cpu_manager.go:414] "containerMap: removing container" podUID="486c40bf-a32e-437c-981e-5eeb29a888f4" containerName="istio-proxy"
Oct 31 00:52:05.817982 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.817874    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486c40bf-a32e-437c-981e-5eeb29a888f4" containerName="istio-proxy"
Oct 31 00:52:05.817982 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.817959    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43eb961-439a-4815-a4e1-8d6e9349fad8" containerName="main"
Oct 31 00:52:05.817982 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.817979    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="486c40bf-a32e-437c-981e-5eeb29a888f4" containerName="istio-proxy"
Oct 31 00:52:05.822411 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.822393    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:05.872736 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.872703    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955537fb-81fd-41ca-abd8-24e2185f9aee-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:05.872865 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.872743    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfrb\" (UniqueName: \"kubernetes.io/projected/955537fb-81fd-41ca-abd8-24e2185f9aee-kube-api-access-rjfrb\") pod \"auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:05.895526 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.895498    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"]
Oct 31 00:52:05.922951 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.922926    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Oct 31 00:52:05.923111 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.922997    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-rz2j9\""
Oct 31 00:52:05.923111 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.923079    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-9cd6c\""
Oct 31 00:52:05.923111 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.923097    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Oct 31 00:52:05.923111 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.923084    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Oct 31 00:52:05.974101 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.974072    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955537fb-81fd-41ca-abd8-24e2185f9aee-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:05.975010 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.974976    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfrb\" (UniqueName: \"kubernetes.io/projected/955537fb-81fd-41ca-abd8-24e2185f9aee-kube-api-access-rjfrb\") pod \"auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:05.982402 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:05.982371    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955537fb-81fd-41ca-abd8-24e2185f9aee-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:06.035178 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:06.035151    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfrb\" (UniqueName: \"kubernetes.io/projected/955537fb-81fd-41ca-abd8-24e2185f9aee-kube-api-access-rjfrb\") pod \"auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:06.134667 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:06.134591    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:06.501337 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:06.501309    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"]
Oct 31 00:52:06.503438 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:52:06.503407    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955537fb_81fd_41ca_abd8_24e2185f9aee.slice/crio-47f88f9a3deb7c4f3adc21cafdc0a39fec712f57c1fbc9b7e223810331ed94ba WatchSource:0}: Error finding container 47f88f9a3deb7c4f3adc21cafdc0a39fec712f57c1fbc9b7e223810331ed94ba: Status 404 returned error can't find the container with id 47f88f9a3deb7c4f3adc21cafdc0a39fec712f57c1fbc9b7e223810331ed94ba
Oct 31 00:52:06.688735 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:06.688696    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" event={"ID":"955537fb-81fd-41ca-abd8-24e2185f9aee","Type":"ContainerStarted","Data":"14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c"}
Oct 31 00:52:06.689132 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:06.688747    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" event={"ID":"955537fb-81fd-41ca-abd8-24e2185f9aee","Type":"ContainerStarted","Data":"47f88f9a3deb7c4f3adc21cafdc0a39fec712f57c1fbc9b7e223810331ed94ba"}
Oct 31 00:52:06.689132 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:06.688810    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:06.801256 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:06.801157    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" podStartSLOduration=1.801142806 podStartE2EDuration="1.801142806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:06.800919125 +0000 UTC m=+1335.355806568" watchObservedRunningTime="2026-10-31 00:52:06.801142806 +0000 UTC m=+1335.356030243"
Oct 31 00:52:18.551522 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:52:18.551490    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:52:37.694934 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:37.694908    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:52:48.552014 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:52:48.551979    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:52:58.288256 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.288222    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"]
Oct 31 00:52:58.291685 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.291665    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.343156 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.343128    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 31 00:52:58.343314 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.343220    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:52:58.343314 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.343258    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 31 00:52:58.388988 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.388959    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 31 00:52:58.389184 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.389037    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-dw7n5\""
Oct 31 00:52:58.389184 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.389060    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 31 00:52:58.391550 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.391529    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"]
Oct 31 00:52:58.397982 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.397953    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.397982 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.397982    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-dshm\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.398162 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.398004    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2949a7b4-e664-493d-b6b0-38b9707137b9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.398162 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.398082    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-home\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.398162 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.398125    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2949a7b4-e664-493d-b6b0-38b9707137b9-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.398321 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.398193    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.398321 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.398214    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gzl\" (UniqueName: \"kubernetes.io/projected/2949a7b4-e664-493d-b6b0-38b9707137b9-kube-api-access-w4gzl\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.398321 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.398249    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.498688 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498657    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.498879 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498695    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-dshm\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.498879 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498725    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2949a7b4-e664-493d-b6b0-38b9707137b9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.498879 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498753    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-home\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.498879 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498788    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2949a7b4-e664-493d-b6b0-38b9707137b9-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.498879 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498835    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.498879 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498860    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gzl\" (UniqueName: \"kubernetes.io/projected/2949a7b4-e664-493d-b6b0-38b9707137b9-kube-api-access-w4gzl\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.499142 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.498909    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.499142 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.499115    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.499241 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.499155    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-home\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.499332 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.499309    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-model-cache\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.499425 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.499377    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.499658 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.499635    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2949a7b4-e664-493d-b6b0-38b9707137b9-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.501114 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.501091    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-dshm\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.501408 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.501391    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2949a7b4-e664-493d-b6b0-38b9707137b9-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.558840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.558770    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gzl\" (UniqueName: \"kubernetes.io/projected/2949a7b4-e664-493d-b6b0-38b9707137b9-kube-api-access-w4gzl\") pod \"tokenizer-clean-path-test-kserve-5bdf556476-jr9dh\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:58.607023 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:58.606999    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:52:59.022178 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:59.022151    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"]
Oct 31 00:52:59.023968 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:52:59.023938    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2949a7b4_e664_493d_b6b0_38b9707137b9.slice/crio-6e04bc8a12e0ad490653773b391f1cac6d70df5dee5eff12769f212ef23c2685 WatchSource:0}: Error finding container 6e04bc8a12e0ad490653773b391f1cac6d70df5dee5eff12769f212ef23c2685: Status 404 returned error can't find the container with id 6e04bc8a12e0ad490653773b391f1cac6d70df5dee5eff12769f212ef23c2685
Oct 31 00:52:59.956942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:59.956904    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" event={"ID":"2949a7b4-e664-493d-b6b0-38b9707137b9","Type":"ContainerStarted","Data":"6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a"}
Oct 31 00:52:59.957328 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:52:59.956984    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" event={"ID":"2949a7b4-e664-493d-b6b0-38b9707137b9","Type":"ContainerStarted","Data":"6e04bc8a12e0ad490653773b391f1cac6d70df5dee5eff12769f212ef23c2685"}
Oct 31 00:53:01.968212 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:01.968180    4888 generic.go:356] "Generic (PLEG): container finished" podID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerID="6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a" exitCode=0
Oct 31 00:53:01.968604 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:01.968256    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" event={"ID":"2949a7b4-e664-493d-b6b0-38b9707137b9","Type":"ContainerDied","Data":"6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a"}
Oct 31 00:53:02.996786 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:02.996751    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" event={"ID":"2949a7b4-e664-493d-b6b0-38b9707137b9","Type":"ContainerStarted","Data":"bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe"}
Oct 31 00:53:03.160994 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:03.160942    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" podStartSLOduration=3.000948062 podStartE2EDuration="5.160927884s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.159979822s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:03.160069611 +0000 UTC m=+1391.714957076" watchObservedRunningTime="2026-10-31 00:53:03.160927884 +0000 UTC m=+1391.715815321"
Oct 31 00:53:03.608109 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:03.608082    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:53:08.607785 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:08.607751    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:53:08.619582 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:08.619558    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:53:09.036015 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:09.035989    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:53:18.551973 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:18.551939    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:53:36.579480 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:36.579450    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"]
Oct 31 00:53:36.580004 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:36.579649    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy" containerID="cri-o://0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316" gracePeriod=30
Oct 31 00:53:36.626392 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:36.626342    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"]
Oct 31 00:53:36.626658 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:36.626609    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy" containerID="cri-o://430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d" gracePeriod=30
Oct 31 00:53:36.628114 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:36.628094    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"]
Oct 31 00:53:36.628301 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:36.628270    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy" containerID="cri-o://24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086" gracePeriod=30
Oct 31 00:53:41.768372 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.766318    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-gxtp4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Oct 31 00:53:41.768372 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.766437    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Oct 31 00:53:41.875493 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.875465    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:53:41.913502 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.913478    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:53:41.950263 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.950239    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:53:41.982616 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982581    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-certs\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.982808 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982623    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-kube-api-access-wgm2v\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-kube-api-access-wgm2v\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.982808 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982656    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-envoy\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.982808 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982713    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-data\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.982808 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982782    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-token\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-token\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.983019 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982813    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2561c6e9-0985-4414-aa04-c4f5bbb68987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2561c6e9-0985-4414-aa04-c4f5bbb68987-istiod-ca-cert\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.983019 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982843    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-socket\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.983019 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982874    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-credential-socket\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.983019 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982938    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-podinfo\") pod \"2561c6e9-0985-4414-aa04-c4f5bbb68987\" (UID: \"2561c6e9-0985-4414-aa04-c4f5bbb68987\") "
Oct 31 00:53:41.983195 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.982868    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-certs" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:41.983195 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983039    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-data" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:41.983195 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983131    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-socket" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:41.983195 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983163    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-credential-socket" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:41.983195 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983190    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2561c6e9-0985-4414-aa04-c4f5bbb68987-istiod-ca-cert" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:41.983433 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983262    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:41.983433 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983282    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:41.983433 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983300    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:41.983433 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.983313    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:41.985036 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.985006    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-token" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:41.985153 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.985062    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-podinfo" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:53:41.985153 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.985083    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-envoy" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:41.985279 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:41.985263    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-kube-api-access-wgm2v" pod "2561c6e9-0985-4414-aa04-c4f5bbb68987" (UID: "2561c6e9-0985-4414-aa04-c4f5bbb68987"). InnerVolumeSpecName "kube-api-access-wgm2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:42.084341 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084312    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-envoy\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084535 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084404    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-envoy\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.084535 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084434    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-kube-api-access-dnlx4\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-kube-api-access-dnlx4\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.084535 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084453    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-socket\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.084535 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084479    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-kube-api-access-6vm5x\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-kube-api-access-6vm5x\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084535 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084506    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-socket\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084542    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istiod-ca-cert\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084583    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3cae128-1e0e-4709-aeed-14737f1f9601-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cae128-1e0e-4709-aeed-14737f1f9601-istiod-ca-cert\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084607    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-token\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-token\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084643    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-podinfo\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084703    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-credential-socket\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084727    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-data\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084754    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-certs\") pod \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\" (UID: \"ab69a17d-f70d-467f-b211-15eb5fd1b05b\") "
Oct 31 00:53:42.084797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084760    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-socket" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084804    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-certs\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084847    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-token\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-token\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084879    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-data\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084909    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-credential-socket\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084940    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-podinfo\") pod \"f3cae128-1e0e-4709-aeed-14737f1f9601\" (UID: \"f3cae128-1e0e-4709-aeed-14737f1f9601\") "
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.084962    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istiod-ca-cert" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085099    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-socket" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.085323 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085246    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-certs" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085543    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3cae128-1e0e-4709-aeed-14737f1f9601-istiod-ca-cert" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085605    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085626    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2561c6e9-0985-4414-aa04-c4f5bbb68987-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085641    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085658    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085673    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085689    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085704    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgm2v\" (UniqueName: \"kubernetes.io/projected/2561c6e9-0985-4414-aa04-c4f5bbb68987-kube-api-access-wgm2v\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085719    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2561c6e9-0985-4414-aa04-c4f5bbb68987-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.085772 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085733    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.086232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085778    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-credential-socket" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.086232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085952    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-credential-socket" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.086232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.085973    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-data" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.086232 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.086119    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-data" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.086465 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.086289    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-certs" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.087392 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.087335    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-kube-api-access-dnlx4" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "kube-api-access-dnlx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:42.087524 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.087393    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-envoy" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.087690 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.087658    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-envoy" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:42.087690 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.087669    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-token" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:42.087862 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.087715    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-podinfo" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:53:42.088470 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.088452    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-token" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:42.088733 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.088720    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-podinfo" pod "f3cae128-1e0e-4709-aeed-14737f1f9601" (UID: "f3cae128-1e0e-4709-aeed-14737f1f9601"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:53:42.089307 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.089289    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-kube-api-access-6vm5x" pod "ab69a17d-f70d-467f-b211-15eb5fd1b05b" (UID: "ab69a17d-f70d-467f-b211-15eb5fd1b05b"). InnerVolumeSpecName "kube-api-access-6vm5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:42.186237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186150    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186181    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186196    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186210    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnlx4\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-kube-api-access-dnlx4\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186226    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vm5x\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-kube-api-access-6vm5x\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186239    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3cae128-1e0e-4709-aeed-14737f1f9601-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186251    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186263    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186277    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186288    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186296    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab69a17d-f70d-467f-b211-15eb5fd1b05b-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186304    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186316    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.186650 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.186324    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3cae128-1e0e-4709-aeed-14737f1f9601-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:42.187102 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.187078    4888 generic.go:356] "Generic (PLEG): container finished" podID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerID="0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316" exitCode=0
Oct 31 00:53:42.187159 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.187123    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" event={"ID":"ab69a17d-f70d-467f-b211-15eb5fd1b05b","Type":"ContainerDied","Data":"0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316"}
Oct 31 00:53:42.187159 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.187142    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"
Oct 31 00:53:42.187260 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.187177    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4" event={"ID":"ab69a17d-f70d-467f-b211-15eb5fd1b05b","Type":"ContainerDied","Data":"f46d2c632c14e0e139a33332790b5fca365d19b06df609eabc0570067399aba8"}
Oct 31 00:53:42.187260 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.187207    4888 scope.go:122] "RemoveContainer" containerID="0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316"
Oct 31 00:53:42.189296 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.189267    4888 generic.go:356] "Generic (PLEG): container finished" podID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerID="430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d" exitCode=0
Oct 31 00:53:42.189447 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.189313    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"
Oct 31 00:53:42.189447 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.189369    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" event={"ID":"2561c6e9-0985-4414-aa04-c4f5bbb68987","Type":"ContainerDied","Data":"430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d"}
Oct 31 00:53:42.189447 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.189411    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s" event={"ID":"2561c6e9-0985-4414-aa04-c4f5bbb68987","Type":"ContainerDied","Data":"05cfb876fd5870d0463baa7099dab403293b2168927cbeb983fc0f98fbc0639c"}
Oct 31 00:53:42.191555 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.191531    4888 generic.go:356] "Generic (PLEG): container finished" podID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerID="24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086" exitCode=0
Oct 31 00:53:42.191691 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.191559    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" event={"ID":"f3cae128-1e0e-4709-aeed-14737f1f9601","Type":"ContainerDied","Data":"24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086"}
Oct 31 00:53:42.191691 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.191598    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"
Oct 31 00:53:42.191691 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.191612    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd" event={"ID":"f3cae128-1e0e-4709-aeed-14737f1f9601","Type":"ContainerDied","Data":"9d2a66780c5673c7884904ab7e6894611ef79f0fb2f91f47384004240ec12ed7"}
Oct 31 00:53:42.196871 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.196856    4888 scope.go:122] "RemoveContainer" containerID="0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316"
Oct 31 00:53:42.197323 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:42.197292    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316\": container with ID starting with 0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316 not found: ID does not exist" containerID="0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316"
Oct 31 00:53:42.197432 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.197332    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316"} err="failed to get container status \"0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316\": rpc error: code = NotFound desc = could not find container \"0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316\": container with ID starting with 0bc87b63d85b1c51552f9ccb6f30a52e59e45f8ed7db788fc07628ec07906316 not found: ID does not exist"
Oct 31 00:53:42.197432 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.197379    4888 scope.go:122] "RemoveContainer" containerID="430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d"
Oct 31 00:53:42.208435 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.208419    4888 scope.go:122] "RemoveContainer" containerID="bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975"
Oct 31 00:53:42.216252 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.216238    4888 scope.go:122] "RemoveContainer" containerID="430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d"
Oct 31 00:53:42.216527 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:42.216507    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d\": container with ID starting with 430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d not found: ID does not exist" containerID="430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d"
Oct 31 00:53:42.216594 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.216533    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d"} err="failed to get container status \"430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d\": rpc error: code = NotFound desc = could not find container \"430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d\": container with ID starting with 430744c18f7c2e578dd819641166a9c90015d5e116f30f9b5bc68c66657fa04d not found: ID does not exist"
Oct 31 00:53:42.216594 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.216551    4888 scope.go:122] "RemoveContainer" containerID="bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975"
Oct 31 00:53:42.216808 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:42.216791    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975\": container with ID starting with bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975 not found: ID does not exist" containerID="bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975"
Oct 31 00:53:42.216842 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.216815    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975"} err="failed to get container status \"bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975\": rpc error: code = NotFound desc = could not find container \"bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975\": container with ID starting with bacd1241ca24a9b01ab9ebe4f270146c6bda4b82581ebf8f6da3033643164975 not found: ID does not exist"
Oct 31 00:53:42.216842 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.216835    4888 scope.go:122] "RemoveContainer" containerID="24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086"
Oct 31 00:53:42.224250 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.224232    4888 scope.go:122] "RemoveContainer" containerID="f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8"
Oct 31 00:53:42.231491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.231465    4888 scope.go:122] "RemoveContainer" containerID="24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086"
Oct 31 00:53:42.231724 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:42.231709    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086\": container with ID starting with 24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086 not found: ID does not exist" containerID="24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086"
Oct 31 00:53:42.231777 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.231731    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086"} err="failed to get container status \"24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086\": rpc error: code = NotFound desc = could not find container \"24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086\": container with ID starting with 24e8b49ab252ce6be169bbac8f7e79bd496b26c9679d3da15a9394e418d70086 not found: ID does not exist"
Oct 31 00:53:42.231777 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.231747    4888 scope.go:122] "RemoveContainer" containerID="f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8"
Oct 31 00:53:42.231954 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:42.231938    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8\": container with ID starting with f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8 not found: ID does not exist" containerID="f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8"
Oct 31 00:53:42.231997 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.231957    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8"} err="failed to get container status \"f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8\": rpc error: code = NotFound desc = could not find container \"f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8\": container with ID starting with f53d956497e96356b0a3985357ac697bc7c90cf14ce71ad627bcb3abf3e53ea8 not found: ID does not exist"
Oct 31 00:53:42.383654 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.383624    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"]
Oct 31 00:53:42.392069 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.392038    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-gxtp4"]
Oct 31 00:53:42.498686 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.498655    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"]
Oct 31 00:53:42.506587 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.506560    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-p857s"]
Oct 31 00:53:42.612645 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.612618    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"]
Oct 31 00:53:42.622137 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:42.622113    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-m76bd"]
Oct 31 00:53:43.260136 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.260107    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"]
Oct 31 00:53:43.269697 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:43.269658    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe" cmd=["/bin/sleep","15"]
Oct 31 00:53:43.269840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.269720    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:53:43.269840 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:43.269737    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerName="main" containerID="cri-o://bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe"
Oct 31 00:53:43.269840 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.269807    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerName="main" containerID="cri-o://bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe" gracePeriod=60
Oct 31 00:53:43.553055 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.553020    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" path="/var/lib/kubelet/pods/2561c6e9-0985-4414-aa04-c4f5bbb68987/volumes"
Oct 31 00:53:43.553886 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.553557    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" path="/var/lib/kubelet/pods/ab69a17d-f70d-467f-b211-15eb5fd1b05b/volumes"
Oct 31 00:53:43.554064 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.554039    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" path="/var/lib/kubelet/pods/f3cae128-1e0e-4709-aeed-14737f1f9601/volumes"
Oct 31 00:53:43.622063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.622040    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:53:43.695851 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.695820    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-tmp-dir\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.695851 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.695852    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-model-cache\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.696085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.695891    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2949a7b4-e664-493d-b6b0-38b9707137b9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2949a7b4-e664-493d-b6b0-38b9707137b9-cabundle-cert\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.696085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.695935    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-dshm\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.696085 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.695953    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-home\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-home\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.696236 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.696120    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-tmp-dir" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.696236 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.696197    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-model-cache" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.696340 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.696262    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-home" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.696340 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.696309    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2949a7b4-e664-493d-b6b0-38b9707137b9-cabundle-cert" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:53:43.698166 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.698145    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-dshm" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.796476 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796393    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2949a7b4-e664-493d-b6b0-38b9707137b9-kube-api-access-w4gzl\" (UniqueName: \"kubernetes.io/projected/2949a7b4-e664-493d-b6b0-38b9707137b9-kube-api-access-w4gzl\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.796476 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796442    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2949a7b4-e664-493d-b6b0-38b9707137b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/2949a7b4-e664-493d-b6b0-38b9707137b9-tls-certs\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.796476 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796470    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-kserve-provision-location\") pod \"2949a7b4-e664-493d-b6b0-38b9707137b9\" (UID: \"2949a7b4-e664-493d-b6b0-38b9707137b9\") "
Oct 31 00:53:43.796701 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796612    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.796701 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796623    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.796701 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796632    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2949a7b4-e664-493d-b6b0-38b9707137b9-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.796701 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796640    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.796701 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.796647    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.798855 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.798795    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2949a7b4-e664-493d-b6b0-38b9707137b9-tls-certs" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:53:43.798855 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.798829    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2949a7b4-e664-493d-b6b0-38b9707137b9-kube-api-access-w4gzl" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "kube-api-access-w4gzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:53:43.817205 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.817174    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-kserve-provision-location" pod "2949a7b4-e664-493d-b6b0-38b9707137b9" (UID: "2949a7b4-e664-493d-b6b0-38b9707137b9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:53:43.897007 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.896977    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4gzl\" (UniqueName: \"kubernetes.io/projected/2949a7b4-e664-493d-b6b0-38b9707137b9-kube-api-access-w4gzl\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.897007 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.897004    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2949a7b4-e664-493d-b6b0-38b9707137b9-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:43.897182 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:43.897013    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2949a7b4-e664-493d-b6b0-38b9707137b9-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:53:44.206722 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.206686    4888 generic.go:356] "Generic (PLEG): container finished" podID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerID="bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe" exitCode=0
Oct 31 00:53:44.206899 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.206737    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"
Oct 31 00:53:44.206899 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.206732    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" event={"ID":"2949a7b4-e664-493d-b6b0-38b9707137b9","Type":"ContainerDied","Data":"bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe"}
Oct 31 00:53:44.206899 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.206858    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh" event={"ID":"2949a7b4-e664-493d-b6b0-38b9707137b9","Type":"ContainerDied","Data":"6e04bc8a12e0ad490653773b391f1cac6d70df5dee5eff12769f212ef23c2685"}
Oct 31 00:53:44.206899 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.206886    4888 scope.go:122] "RemoveContainer" containerID="bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe"
Oct 31 00:53:44.218223 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.218202    4888 scope.go:122] "RemoveContainer" containerID="6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a"
Oct 31 00:53:44.226280 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.226264    4888 scope.go:122] "RemoveContainer" containerID="bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe"
Oct 31 00:53:44.226570 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:44.226544    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe\": container with ID starting with bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe not found: ID does not exist" containerID="bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe"
Oct 31 00:53:44.226607 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.226581    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe"} err="failed to get container status \"bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe\": rpc error: code = NotFound desc = could not find container \"bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe\": container with ID starting with bb7874c4dadf1e740ea2739fa28441b1aba8a7cf9b0eb36bc1ff35bb75264ebe not found: ID does not exist"
Oct 31 00:53:44.226607 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.226599    4888 scope.go:122] "RemoveContainer" containerID="6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a"
Oct 31 00:53:44.226824 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:44.226810    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a\": container with ID starting with 6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a not found: ID does not exist" containerID="6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a"
Oct 31 00:53:44.226867 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.226828    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a"} err="failed to get container status \"6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a\": rpc error: code = NotFound desc = could not find container \"6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a\": container with ID starting with 6d11ea341652870512c5202445ee7fa445d9f465d629e84c740136ac9ba3928a not found: ID does not exist"
Oct 31 00:53:44.371696 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.371662    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"]
Oct 31 00:53:44.417522 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:44.417488    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5bdf556476-jr9dh"]
Oct 31 00:53:45.553677 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:53:45.553646    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" path="/var/lib/kubelet/pods/2949a7b4-e664-493d-b6b0-38b9707137b9/volumes"
Oct 31 00:53:48.551640 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:53:48.551601    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:54:18.552129 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:54:18.552089    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:54:35.821268 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:35.821234    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"]
Oct 31 00:54:36.180574 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180499    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"]
Oct 31 00:54:36.180830 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180819    4888 cpu_manager.go:414] "containerMap: removing container" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerName="storage-initializer"
Oct 31 00:54:36.180872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180832    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerName="storage-initializer"
Oct 31 00:54:36.180872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180845    4888 cpu_manager.go:414] "containerMap: removing container" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy"
Oct 31 00:54:36.180872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180850    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy"
Oct 31 00:54:36.180872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180857    4888 cpu_manager.go:414] "containerMap: removing container" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy"
Oct 31 00:54:36.180872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180862    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy"
Oct 31 00:54:36.180872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180869    4888 cpu_manager.go:414] "containerMap: removing container" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerName="main"
Oct 31 00:54:36.180872 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180875    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerName="main"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180888    4888 cpu_manager.go:414] "containerMap: removing container" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180894    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180900    4888 cpu_manager.go:414] "containerMap: removing container" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180904    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180910    4888 cpu_manager.go:414] "containerMap: removing container" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180914    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180964    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180970    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="2561c6e9-0985-4414-aa04-c4f5bbb68987" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180977    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab69a17d-f70d-467f-b211-15eb5fd1b05b" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180985    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180990    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3cae128-1e0e-4709-aeed-14737f1f9601" containerName="istio-proxy"
Oct 31 00:54:36.181073 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.180994    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="2949a7b4-e664-493d-b6b0-38b9707137b9" containerName="main"
Oct 31 00:54:36.184047 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.184030    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.235180 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.235150    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:54:36.235180 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.235168    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 31 00:54:36.235180 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.235178    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 31 00:54:36.242534 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.242504    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"]
Oct 31 00:54:36.288542 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.286019    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 31 00:54:36.288542 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.286032    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-5nnvq\""
Oct 31 00:54:36.288542 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.286132    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 31 00:54:36.309849 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.309824    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-home\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.310012 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.309857    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69dc493d-f593-4892-8c30-410065f226a0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.310012 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.309878    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-model-cache\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.310012 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.309970    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.310012 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.309990    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dc493d-f593-4892-8c30-410065f226a0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.310168 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.310029    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg8qw\" (UniqueName: \"kubernetes.io/projected/69dc493d-f593-4892-8c30-410065f226a0-kube-api-access-jg8qw\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.310201 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.310151    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-dshm\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.310235 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.310194    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.411333 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411298    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-home\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.411333 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411384    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69dc493d-f593-4892-8c30-410065f226a0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.413196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411415    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-model-cache\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.413196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411451    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.413196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411472    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dc493d-f593-4892-8c30-410065f226a0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.413196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411524    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg8qw\" (UniqueName: \"kubernetes.io/projected/69dc493d-f593-4892-8c30-410065f226a0-kube-api-access-jg8qw\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.413196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411554    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-dshm\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.413196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.411578    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.413196 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.413049    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.416258 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.413508    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-home\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.416258 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.413928    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-model-cache\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.416258 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.414027    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.416258 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.414304    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69dc493d-f593-4892-8c30-410065f226a0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.417049 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.417030    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dc493d-f593-4892-8c30-410065f226a0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.418007 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.417988    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-dshm\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.474613 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.474587    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg8qw\" (UniqueName: \"kubernetes.io/projected/69dc493d-f593-4892-8c30-410065f226a0-kube-api-access-jg8qw\") pod \"tokenizer-migration-path-test-kserve-56b89f5459-b8pqd\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.499209 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.499175    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:36.926482 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.926411    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"]
Oct 31 00:54:36.934436 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:54:36.934412    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69dc493d_f593_4892_8c30_410065f226a0.slice/crio-1eaa05e16a984b74403b3da85611fa5598d1fe27c76fb93c5e042b9f9d4c89c1 WatchSource:0}: Error finding container 1eaa05e16a984b74403b3da85611fa5598d1fe27c76fb93c5e042b9f9d4c89c1: Status 404 returned error can't find the container with id 1eaa05e16a984b74403b3da85611fa5598d1fe27c76fb93c5e042b9f9d4c89c1
Oct 31 00:54:36.985261 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.985232    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"]
Oct 31 00:54:36.988795 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:36.988774    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.039683 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.039618    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-n2bjt\""
Oct 31 00:54:37.046551 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.046521    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"]
Oct 31 00:54:37.117594 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.117563    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.117782 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.117707    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xxb2\" (UniqueName: \"kubernetes.io/projected/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-kube-api-access-8xxb2\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.218925 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.218879    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.219131 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.218963    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xxb2\" (UniqueName: \"kubernetes.io/projected/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-kube-api-access-8xxb2\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.221475 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.221454    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.279069 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.279032    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xxb2\" (UniqueName: \"kubernetes.io/projected/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-kube-api-access-8xxb2\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.306407 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.306381    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:37.443777 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.443746    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" event={"ID":"69dc493d-f593-4892-8c30-410065f226a0","Type":"ContainerStarted","Data":"85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11"}
Oct 31 00:54:37.443938 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.443793    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" event={"ID":"69dc493d-f593-4892-8c30-410065f226a0","Type":"ContainerStarted","Data":"1eaa05e16a984b74403b3da85611fa5598d1fe27c76fb93c5e042b9f9d4c89c1"}
Oct 31 00:54:37.680237 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.680159    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"]
Oct 31 00:54:37.687994 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:54:37.687960    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0075c8_3f64_4a71_aec1_539ef7e26b6c.slice/crio-fea86ef8edf9e823ad4b0218518a4426b65009c096686bad6d6ec123dc795e18 WatchSource:0}: Error finding container fea86ef8edf9e823ad4b0218518a4426b65009c096686bad6d6ec123dc795e18: Status 404 returned error can't find the container with id fea86ef8edf9e823ad4b0218518a4426b65009c096686bad6d6ec123dc795e18
Oct 31 00:54:37.695462 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:37.695425    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:54:38.459508 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:38.459471    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" event={"ID":"5b0075c8-3f64-4a71-aec1-539ef7e26b6c","Type":"ContainerStarted","Data":"2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86"}
Oct 31 00:54:38.460063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:38.459543    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" event={"ID":"5b0075c8-3f64-4a71-aec1-539ef7e26b6c","Type":"ContainerStarted","Data":"fea86ef8edf9e823ad4b0218518a4426b65009c096686bad6d6ec123dc795e18"}
Oct 31 00:54:38.460063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:38.459950    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:54:38.572907 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:38.572856    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" podStartSLOduration=2.572842992 podStartE2EDuration="2.572842992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:38.571821195 +0000 UTC m=+1487.126708633" watchObservedRunningTime="2026-10-31 00:54:38.572842992 +0000 UTC m=+1487.127730430"
Oct 31 00:54:39.465534 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:39.465503    4888 generic.go:356] "Generic (PLEG): container finished" podID="69dc493d-f593-4892-8c30-410065f226a0" containerID="85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11" exitCode=0
Oct 31 00:54:39.465986 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:39.465580    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" event={"ID":"69dc493d-f593-4892-8c30-410065f226a0","Type":"ContainerDied","Data":"85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11"}
Oct 31 00:54:40.472231 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:40.472201    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" event={"ID":"69dc493d-f593-4892-8c30-410065f226a0","Type":"ContainerStarted","Data":"0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b"}
Oct 31 00:54:40.499509 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:40.499475    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:40.683410 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:40.683342    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" podStartSLOduration=2.609826619 podStartE2EDuration="4.683328366s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.073501747s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:54:40.635384033 +0000 UTC m=+1489.190271468" watchObservedRunningTime="2026-10-31 00:54:40.683328366 +0000 UTC m=+1489.238215799"
Oct 31 00:54:46.499950 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:46.499910    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:46.512198 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:46.512170    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:47.510757 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:47.510727    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:54:47.697087 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:47.697026    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:54:48.551851 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:54:48.551815    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:54:50.846284 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:50.846248    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main" containerID="cri-o://14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c" gracePeriod=45
Oct 31 00:54:51.707337 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:51.707305    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:54:51.710163 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:51.710140    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:54:57.695621 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:57.695587    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:54:57.696066 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:54:57.695694    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:55:07.695440 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:07.695402    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:55:09.467492 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:09.467465    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:55:17.695507 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:17.695464    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:55:18.551844 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:55:18.551802    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:55:21.092032 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.092006    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:55:21.180456 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.180372    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/955537fb-81fd-41ca-abd8-24e2185f9aee-kube-api-access-rjfrb\" (UniqueName: \"kubernetes.io/projected/955537fb-81fd-41ca-abd8-24e2185f9aee-kube-api-access-rjfrb\") pod \"955537fb-81fd-41ca-abd8-24e2185f9aee\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") "
Oct 31 00:55:21.180620 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.180513    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/955537fb-81fd-41ca-abd8-24e2185f9aee-tls-certs\" (UniqueName: \"kubernetes.io/secret/955537fb-81fd-41ca-abd8-24e2185f9aee-tls-certs\") pod \"955537fb-81fd-41ca-abd8-24e2185f9aee\" (UID: \"955537fb-81fd-41ca-abd8-24e2185f9aee\") "
Oct 31 00:55:21.182531 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.182501    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955537fb-81fd-41ca-abd8-24e2185f9aee-kube-api-access-rjfrb" pod "955537fb-81fd-41ca-abd8-24e2185f9aee" (UID: "955537fb-81fd-41ca-abd8-24e2185f9aee"). InnerVolumeSpecName "kube-api-access-rjfrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:21.182531 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.182525    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/955537fb-81fd-41ca-abd8-24e2185f9aee-tls-certs" pod "955537fb-81fd-41ca-abd8-24e2185f9aee" (UID: "955537fb-81fd-41ca-abd8-24e2185f9aee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:21.239755 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.239720    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"]
Oct 31 00:55:21.273457 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.273431    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"]
Oct 31 00:55:21.282006 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:55:21.281973    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b" cmd=["/bin/sleep","15"]
Oct 31 00:55:21.282156 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.282018    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjfrb\" (UniqueName: \"kubernetes.io/projected/955537fb-81fd-41ca-abd8-24e2185f9aee-kube-api-access-rjfrb\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.282156 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.282045    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/955537fb-81fd-41ca-abd8-24e2185f9aee-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.282156 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.282024    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:55:21.282156 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:55:21.282061    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="main" containerID="cri-o://0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b"
Oct 31 00:55:21.282156 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.282097    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="main" containerID="cri-o://0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b" gracePeriod=60
Oct 31 00:55:21.511865 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.511828    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": dial tcp 10.134.0.47:8000: connect: connection refused"
Oct 31 00:55:21.611745 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.611722    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:55:21.653199 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.653167    4888 generic.go:356] "Generic (PLEG): container finished" podID="69dc493d-f593-4892-8c30-410065f226a0" containerID="0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b" exitCode=0
Oct 31 00:55:21.653340 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.653210    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"
Oct 31 00:55:21.653340 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.653244    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" event={"ID":"69dc493d-f593-4892-8c30-410065f226a0","Type":"ContainerDied","Data":"0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b"}
Oct 31 00:55:21.653340 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.653294    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd" event={"ID":"69dc493d-f593-4892-8c30-410065f226a0","Type":"ContainerDied","Data":"1eaa05e16a984b74403b3da85611fa5598d1fe27c76fb93c5e042b9f9d4c89c1"}
Oct 31 00:55:21.653340 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.653315    4888 scope.go:122] "RemoveContainer" containerID="0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b"
Oct 31 00:55:21.654490 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.654471    4888 generic.go:356] "Generic (PLEG): container finished" podID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerID="14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c" exitCode=0
Oct 31 00:55:21.654593 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.654528    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"
Oct 31 00:55:21.654748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.654670    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" event={"ID":"955537fb-81fd-41ca-abd8-24e2185f9aee","Type":"ContainerDied","Data":"14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c"}
Oct 31 00:55:21.654823 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.654782    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49" event={"ID":"955537fb-81fd-41ca-abd8-24e2185f9aee","Type":"ContainerDied","Data":"47f88f9a3deb7c4f3adc21cafdc0a39fec712f57c1fbc9b7e223810331ed94ba"}
Oct 31 00:55:21.661797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.661780    4888 scope.go:122] "RemoveContainer" containerID="85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11"
Oct 31 00:55:21.669278 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.669257    4888 scope.go:122] "RemoveContainer" containerID="0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b"
Oct 31 00:55:21.669568 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:55:21.669548    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b\": container with ID starting with 0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b not found: ID does not exist" containerID="0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b"
Oct 31 00:55:21.669639 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.669576    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b"} err="failed to get container status \"0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b\": rpc error: code = NotFound desc = could not find container \"0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b\": container with ID starting with 0b72ba13bd6fd33d16127ed3bc06874d909861dcb501fcd6360ffe59226fe44b not found: ID does not exist"
Oct 31 00:55:21.669639 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.669593    4888 scope.go:122] "RemoveContainer" containerID="85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11"
Oct 31 00:55:21.669845 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:55:21.669828    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11\": container with ID starting with 85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11 not found: ID does not exist" containerID="85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11"
Oct 31 00:55:21.669891 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.669852    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11"} err="failed to get container status \"85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11\": rpc error: code = NotFound desc = could not find container \"85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11\": container with ID starting with 85bd7f668f575a3f98ff793a7fc2c575566f9e6c56834dfd2fee740c7d771e11 not found: ID does not exist"
Oct 31 00:55:21.669891 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.669871    4888 scope.go:122] "RemoveContainer" containerID="14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c"
Oct 31 00:55:21.677067 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.677047    4888 scope.go:122] "RemoveContainer" containerID="14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c"
Oct 31 00:55:21.677334 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:55:21.677316    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c\": container with ID starting with 14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c not found: ID does not exist" containerID="14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c"
Oct 31 00:55:21.677417 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.677344    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c"} err="failed to get container status \"14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c\": rpc error: code = NotFound desc = could not find container \"14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c\": container with ID starting with 14a729dd80a53fc5aed4cd9ff8b88c18d88b2dfda8144958501c0493c0588e9c not found: ID does not exist"
Oct 31 00:55:21.686661 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686643    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69dc493d-f593-4892-8c30-410065f226a0-kube-api-access-jg8qw\" (UniqueName: \"kubernetes.io/projected/69dc493d-f593-4892-8c30-410065f226a0-kube-api-access-jg8qw\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.686743 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686683    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-home\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-home\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.686814 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686770    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-model-cache\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.686866 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686835    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-dshm\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.686918 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686865    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69dc493d-f593-4892-8c30-410065f226a0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69dc493d-f593-4892-8c30-410065f226a0-cabundle-cert\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.686918 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686894    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-kserve-provision-location\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.687025 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686945    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-tmp-dir\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.687025 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686947    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-home" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.687025 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.686972    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69dc493d-f593-4892-8c30-410065f226a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/69dc493d-f593-4892-8c30-410065f226a0-tls-certs\") pod \"69dc493d-f593-4892-8c30-410065f226a0\" (UID: \"69dc493d-f593-4892-8c30-410065f226a0\") "
Oct 31 00:55:21.687025 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.687011    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-model-cache" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.687295 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.687231    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.687400 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.687237    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-tmp-dir" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.687400 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.687276    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69dc493d-f593-4892-8c30-410065f226a0-cabundle-cert" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:55:21.687498 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.687402    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.689016 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.688996    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69dc493d-f593-4892-8c30-410065f226a0-kube-api-access-jg8qw" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "kube-api-access-jg8qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:55:21.689077 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.689040    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69dc493d-f593-4892-8c30-410065f226a0-tls-certs" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:55:21.689157 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.689140    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-dshm" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.708173 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.708151    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-kserve-provision-location" pod "69dc493d-f593-4892-8c30-410065f226a0" (UID: "69dc493d-f593-4892-8c30-410065f226a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:55:21.787868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.787796    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.787868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.787823    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69dc493d-f593-4892-8c30-410065f226a0-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.787868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.787835    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg8qw\" (UniqueName: \"kubernetes.io/projected/69dc493d-f593-4892-8c30-410065f226a0-kube-api-access-jg8qw\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.787868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.787844    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.787868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.787853    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69dc493d-f593-4892-8c30-410065f226a0-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.787868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.787861    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69dc493d-f593-4892-8c30-410065f226a0-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:55:21.888063 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.888031    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"]
Oct 31 00:55:21.894576 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:21.894550    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-674b8746cc-8zh49"]
Oct 31 00:55:22.071706 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:22.071635    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"]
Oct 31 00:55:22.077843 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:22.077818    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-56b89f5459-b8pqd"]
Oct 31 00:55:23.553079 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:23.553047    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69dc493d-f593-4892-8c30-410065f226a0" path="/var/lib/kubelet/pods/69dc493d-f593-4892-8c30-410065f226a0/volumes"
Oct 31 00:55:23.553481 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:23.553430    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" path="/var/lib/kubelet/pods/955537fb-81fd-41ca-abd8-24e2185f9aee/volumes"
Oct 31 00:55:29.072559 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072526    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"]
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072839    4888 cpu_manager.go:414] "containerMap: removing container" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="main"
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072850    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="main"
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072870    4888 cpu_manager.go:414] "containerMap: removing container" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main"
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072875    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main"
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072890    4888 cpu_manager.go:414] "containerMap: removing container" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="storage-initializer"
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072896    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="storage-initializer"
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072945    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="69dc493d-f593-4892-8c30-410065f226a0" containerName="main"
Oct 31 00:55:29.073020 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.072953    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="955537fb-81fd-41ca-abd8-24e2185f9aee" containerName="main"
Oct 31 00:55:29.077382 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.077361    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.129704 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.129670    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Oct 31 00:55:29.150022 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.149995    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgpk\" (UniqueName: \"kubernetes.io/projected/528ceaae-729e-4097-9e95-fa718424e470-kube-api-access-vfgpk\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.150189 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.150034    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528ceaae-729e-4097-9e95-fa718424e470-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.162815 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.162794    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"]
Oct 31 00:55:29.177797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.177764    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-p5g7p\""
Oct 31 00:55:29.177797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.177781    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-gttt6\""
Oct 31 00:55:29.177797 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.177765    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Oct 31 00:55:29.229178 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.229156    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Oct 31 00:55:29.251336 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.251302    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgpk\" (UniqueName: \"kubernetes.io/projected/528ceaae-729e-4097-9e95-fa718424e470-kube-api-access-vfgpk\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.251336 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.251368    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528ceaae-729e-4097-9e95-fa718424e470-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.254167 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.254141    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528ceaae-729e-4097-9e95-fa718424e470-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.312060 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.312035    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgpk\" (UniqueName: \"kubernetes.io/projected/528ceaae-729e-4097-9e95-fa718424e470-kube-api-access-vfgpk\") pod \"auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.389804 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.389738    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:29.467252 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.467218    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:55:29.770307 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:29.770274    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"]
Oct 31 00:55:29.777266 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:55:29.777230    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528ceaae_729e_4097_9e95_fa718424e470.slice/crio-20c35be2fe17aa284305d7a2795832ba84c08255a9ee118a429b67c886e0b23b WatchSource:0}: Error finding container 20c35be2fe17aa284305d7a2795832ba84c08255a9ee118a429b67c886e0b23b: Status 404 returned error can't find the container with id 20c35be2fe17aa284305d7a2795832ba84c08255a9ee118a429b67c886e0b23b
Oct 31 00:55:30.698402 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:30.698369    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" event={"ID":"528ceaae-729e-4097-9e95-fa718424e470","Type":"ContainerStarted","Data":"01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4"}
Oct 31 00:55:30.698829 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:30.698457    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" event={"ID":"528ceaae-729e-4097-9e95-fa718424e470","Type":"ContainerStarted","Data":"20c35be2fe17aa284305d7a2795832ba84c08255a9ee118a429b67c886e0b23b"}
Oct 31 00:55:30.698829 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:30.698483    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:55:30.811212 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:30.811168    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" podStartSLOduration=1.811155691 podStartE2EDuration="1.811155691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:30.810454809 +0000 UTC m=+1539.365342245" watchObservedRunningTime="2026-10-31 00:55:30.811155691 +0000 UTC m=+1539.366043123"
Oct 31 00:55:36.259090 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:36.259017    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main" containerID="cri-o://2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86" gracePeriod=45
Oct 31 00:55:39.467923 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:39.467885    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:55:48.551590 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:55:48.551554    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:55:49.468018 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:49.467935    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:55:49.468165 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:49.468043    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:55:59.467047 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:55:59.467007    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:56:01.705823 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:01.705796    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:56:06.497606 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.497583    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:56:06.541595 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.541536    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-kube-api-access-8xxb2\" (UniqueName: \"kubernetes.io/projected/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-kube-api-access-8xxb2\") pod \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") "
Oct 31 00:56:06.541595 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.541594    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-tls-certs\") pod \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\" (UID: \"5b0075c8-3f64-4a71-aec1-539ef7e26b6c\") "
Oct 31 00:56:06.543642 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.543617    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-kube-api-access-8xxb2" pod "5b0075c8-3f64-4a71-aec1-539ef7e26b6c" (UID: "5b0075c8-3f64-4a71-aec1-539ef7e26b6c"). InnerVolumeSpecName "kube-api-access-8xxb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:56:06.543727 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.543651    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-tls-certs" pod "5b0075c8-3f64-4a71-aec1-539ef7e26b6c" (UID: "5b0075c8-3f64-4a71-aec1-539ef7e26b6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:56:06.642743 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.642713    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xxb2\" (UniqueName: \"kubernetes.io/projected/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-kube-api-access-8xxb2\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:56:06.642743 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.642738    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0075c8-3f64-4a71-aec1-539ef7e26b6c-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:56:06.859759 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.859684    4888 generic.go:356] "Generic (PLEG): container finished" podID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerID="2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86" exitCode=0
Oct 31 00:56:06.859759 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.859725    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"
Oct 31 00:56:06.859927 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.859760    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" event={"ID":"5b0075c8-3f64-4a71-aec1-539ef7e26b6c","Type":"ContainerDied","Data":"2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86"}
Oct 31 00:56:06.859927 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.859811    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8" event={"ID":"5b0075c8-3f64-4a71-aec1-539ef7e26b6c","Type":"ContainerDied","Data":"fea86ef8edf9e823ad4b0218518a4426b65009c096686bad6d6ec123dc795e18"}
Oct 31 00:56:06.859927 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.859832    4888 scope.go:122] "RemoveContainer" containerID="2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86"
Oct 31 00:56:06.868033 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.868018    4888 scope.go:122] "RemoveContainer" containerID="2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86"
Oct 31 00:56:06.868295 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:56:06.868275    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86\": container with ID starting with 2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86 not found: ID does not exist" containerID="2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86"
Oct 31 00:56:06.868377 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:06.868304    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86"} err="failed to get container status \"2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86\": rpc error: code = NotFound desc = could not find container \"2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86\": container with ID starting with 2259274ee04658102f50f5b24e6ce86c9a8dbe201d60d24d3e1d108047271a86 not found: ID does not exist"
Oct 31 00:56:07.021885 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:07.021853    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"]
Oct 31 00:56:07.068061 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:07.068037    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5666bzjw8"]
Oct 31 00:56:07.552982 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:07.552941    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" path="/var/lib/kubelet/pods/5b0075c8-3f64-4a71-aec1-539ef7e26b6c/volumes"
Oct 31 00:56:16.795751 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.795719    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"]
Oct 31 00:56:16.796154 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.796026    4888 cpu_manager.go:414] "containerMap: removing container" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main"
Oct 31 00:56:16.796154 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.796036    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main"
Oct 31 00:56:16.796154 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.796106    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0075c8-3f64-4a71-aec1-539ef7e26b6c" containerName="main"
Oct 31 00:56:16.800608 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.800584    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.853264 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.853236    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 31 00:56:16.899438 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.899413    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 31 00:56:16.899438 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.899412    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 31 00:56:16.899438 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.899413    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-cdn7h\""
Oct 31 00:56:16.899438 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.899435    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 31 00:56:16.899750 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.899458    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:56:16.901197 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.901173    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"]
Oct 31 00:56:16.923848 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.923819    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.923998 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.923864    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d65fe27e-a6c2-40b1-9656-0dc6a280123d-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.923998 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.923882    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tls-certs\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.923998 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.923898    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hm5z\" (UniqueName: \"kubernetes.io/projected/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kube-api-access-7hm5z\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.924115 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.924083    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.924154 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.924137    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-home\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.924206 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.924185    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-model-cache\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:16.924261 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:16.924211    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-dshm\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.024903 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.024866    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.025095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.024925    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d65fe27e-a6c2-40b1-9656-0dc6a280123d-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.025095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.024955    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tls-certs\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.025095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.024978    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hm5z\" (UniqueName: \"kubernetes.io/projected/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kube-api-access-7hm5z\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.025369 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.025325    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tmp-dir\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026108 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.025325    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026108 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.025784    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-home\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026108 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.025808    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-model-cache\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026108 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.025826    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d65fe27e-a6c2-40b1-9656-0dc6a280123d-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026108 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.025836    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-dshm\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026108 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.026064    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-model-cache\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026445 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.026332    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-home\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.026445 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.025584    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.028910 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.028891    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-dshm\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.029145 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.029128    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tls-certs\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.085558 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.085483    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hm5z\" (UniqueName: \"kubernetes.io/projected/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kube-api-access-7hm5z\") pod \"tokenizer-template-args-test-kserve-7995846755-fx8j2\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.114594 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.114563    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:17.546884 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.546853    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"]
Oct 31 00:56:17.555521 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:56:17.555477    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65fe27e_a6c2_40b1_9656_0dc6a280123d.slice/crio-90479d936a1bdcde1b010c63f1cb21fc5f64af1f5307af7b30917641ea79b04e WatchSource:0}: Error finding container 90479d936a1bdcde1b010c63f1cb21fc5f64af1f5307af7b30917641ea79b04e: Status 404 returned error can't find the container with id 90479d936a1bdcde1b010c63f1cb21fc5f64af1f5307af7b30917641ea79b04e
Oct 31 00:56:17.915854 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.915767    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" event={"ID":"d65fe27e-a6c2-40b1-9656-0dc6a280123d","Type":"ContainerStarted","Data":"ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e"}
Oct 31 00:56:17.916231 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:17.915866    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" event={"ID":"d65fe27e-a6c2-40b1-9656-0dc6a280123d","Type":"ContainerStarted","Data":"90479d936a1bdcde1b010c63f1cb21fc5f64af1f5307af7b30917641ea79b04e"}
Oct 31 00:56:18.555189 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:56:18.555152    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:56:19.928391 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:19.928339    4888 generic.go:356] "Generic (PLEG): container finished" podID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerID="ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e" exitCode=0
Oct 31 00:56:19.928795 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:19.928416    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" event={"ID":"d65fe27e-a6c2-40b1-9656-0dc6a280123d","Type":"ContainerDied","Data":"ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e"}
Oct 31 00:56:20.934571 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:20.934536    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" event={"ID":"d65fe27e-a6c2-40b1-9656-0dc6a280123d","Type":"ContainerStarted","Data":"3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039"}
Oct 31 00:56:21.109157 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:21.109099    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" podStartSLOduration=3.018905938 podStartE2EDuration="5.10908458s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.090178642s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:56:21.107555926 +0000 UTC m=+1589.662443378" watchObservedRunningTime="2026-10-31 00:56:21.10908458 +0000 UTC m=+1589.663972018"
Oct 31 00:56:21.115637 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:21.115610    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:27.114894 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:27.114858    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:27.129509 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:27.129484    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:27.977968 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:56:27.977938    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:56:48.551852 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:56:48.551810    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:57:00.972113 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:00.972079    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"]
Oct 31 00:57:00.981603 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:57:00.981560    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039" cmd=["/bin/sleep","15"]
Oct 31 00:57:00.981746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:00.981612    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 00:57:00.981746 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:57:00.981626    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerName="main" containerID="cri-o://3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039"
Oct 31 00:57:00.981746 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:00.981666    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerName="main" containerID="cri-o://3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039" gracePeriod=60
Oct 31 00:57:01.328971 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.328946    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:57:01.384818 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.384794    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65fe27e-a6c2-40b1-9656-0dc6a280123d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d65fe27e-a6c2-40b1-9656-0dc6a280123d-cabundle-cert\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.384942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.384827    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-model-cache\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.384942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.384845    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-dshm\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.384942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.384865    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kserve-provision-location\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.384942 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.384900    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kube-api-access-7hm5z\" (UniqueName: \"kubernetes.io/projected/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kube-api-access-7hm5z\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.385123 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.384988    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tmp-dir\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.385123 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385042    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tls-certs\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.385224 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385122    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-home\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-home\") pod \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\" (UID: \"d65fe27e-a6c2-40b1-9656-0dc6a280123d\") "
Oct 31 00:57:01.385224 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385139    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-model-cache" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.385330 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385251    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65fe27e-a6c2-40b1-9656-0dc6a280123d-cabundle-cert" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:57:01.385413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385315    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tmp-dir" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.385495 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385429    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.385572 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385503    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d65fe27e-a6c2-40b1-9656-0dc6a280123d-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.385572 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385507    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-home" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.385572 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.385520    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.387227 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.387206    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-dshm" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.387324 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.387261    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kube-api-access-7hm5z" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "kube-api-access-7hm5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:57:01.387479 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.387462    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tls-certs" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:57:01.405781 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.405758    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kserve-provision-location" pod "d65fe27e-a6c2-40b1-9656-0dc6a280123d" (UID: "d65fe27e-a6c2-40b1-9656-0dc6a280123d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:57:01.486477 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.486452    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.486477 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.486477    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.486634 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.486487    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hm5z\" (UniqueName: \"kubernetes.io/projected/d65fe27e-a6c2-40b1-9656-0dc6a280123d-kube-api-access-7hm5z\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.486634 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.486495    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d65fe27e-a6c2-40b1-9656-0dc6a280123d-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:01.486634 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:01.486503    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d65fe27e-a6c2-40b1-9656-0dc6a280123d-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:57:02.138079 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.138041    4888 generic.go:356] "Generic (PLEG): container finished" podID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerID="3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039" exitCode=0
Oct 31 00:57:02.138553 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.138093    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"
Oct 31 00:57:02.138553 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.138110    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" event={"ID":"d65fe27e-a6c2-40b1-9656-0dc6a280123d","Type":"ContainerDied","Data":"3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039"}
Oct 31 00:57:02.138553 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.138167    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2" event={"ID":"d65fe27e-a6c2-40b1-9656-0dc6a280123d","Type":"ContainerDied","Data":"90479d936a1bdcde1b010c63f1cb21fc5f64af1f5307af7b30917641ea79b04e"}
Oct 31 00:57:02.138553 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.138201    4888 scope.go:122] "RemoveContainer" containerID="3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039"
Oct 31 00:57:02.146980 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.146963    4888 scope.go:122] "RemoveContainer" containerID="ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e"
Oct 31 00:57:02.154413 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.154399    4888 scope.go:122] "RemoveContainer" containerID="3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039"
Oct 31 00:57:02.154689 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:57:02.154670    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039\": container with ID starting with 3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039 not found: ID does not exist" containerID="3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039"
Oct 31 00:57:02.154738 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.154697    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039"} err="failed to get container status \"3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039\": rpc error: code = NotFound desc = could not find container \"3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039\": container with ID starting with 3c826501d4d93192616f8c077d04408859e72756aa94fd130e17e35e78ceb039 not found: ID does not exist"
Oct 31 00:57:02.154738 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.154713    4888 scope.go:122] "RemoveContainer" containerID="ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e"
Oct 31 00:57:02.154989 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:57:02.154970    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e\": container with ID starting with ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e not found: ID does not exist" containerID="ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e"
Oct 31 00:57:02.155041 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.154995    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e"} err="failed to get container status \"ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e\": rpc error: code = NotFound desc = could not find container \"ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e\": container with ID starting with ceea0329243f9b92ce7077e6940f7f3e15840b07c235d5c1c1a1e1b1bc48c56e not found: ID does not exist"
Oct 31 00:57:02.344521 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.344491    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"]
Oct 31 00:57:02.352535 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:02.352507    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-7995846755-fx8j2"]
Oct 31 00:57:03.556058 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:03.556027    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" path="/var/lib/kubelet/pods/d65fe27e-a6c2-40b1-9656-0dc6a280123d/volumes"
Oct 31 00:57:18.551990 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:57:18.551954    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:57:26.804307 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:26.804224    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"]
Oct 31 00:57:31.705835 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:31.705798    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:57:41.705149 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:41.705115    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:57:41.823198 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:41.823161    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main" containerID="cri-o://01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4" gracePeriod=45
Oct 31 00:57:48.551547 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:57:48.551512    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:57:51.705304 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:51.705272    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:57:51.705723 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:57:51.705409    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:58:01.705748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:01.705704    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:58:11.705668 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:11.705627    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:58:12.072141 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.072117    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:58:12.143873 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.143844    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528ceaae-729e-4097-9e95-fa718424e470-kube-api-access-vfgpk\" (UniqueName: \"kubernetes.io/projected/528ceaae-729e-4097-9e95-fa718424e470-kube-api-access-vfgpk\") pod \"528ceaae-729e-4097-9e95-fa718424e470\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") "
Oct 31 00:58:12.144022 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.143886    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528ceaae-729e-4097-9e95-fa718424e470-tls-certs\" (UniqueName: \"kubernetes.io/secret/528ceaae-729e-4097-9e95-fa718424e470-tls-certs\") pod \"528ceaae-729e-4097-9e95-fa718424e470\" (UID: \"528ceaae-729e-4097-9e95-fa718424e470\") "
Oct 31 00:58:12.146164 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.146139    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528ceaae-729e-4097-9e95-fa718424e470-kube-api-access-vfgpk" pod "528ceaae-729e-4097-9e95-fa718424e470" (UID: "528ceaae-729e-4097-9e95-fa718424e470"). InnerVolumeSpecName "kube-api-access-vfgpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:58:12.146249 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.146175    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528ceaae-729e-4097-9e95-fa718424e470-tls-certs" pod "528ceaae-729e-4097-9e95-fa718424e470" (UID: "528ceaae-729e-4097-9e95-fa718424e470"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:58:12.244668 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.244584    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfgpk\" (UniqueName: \"kubernetes.io/projected/528ceaae-729e-4097-9e95-fa718424e470-kube-api-access-vfgpk\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:12.244668 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.244614    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/528ceaae-729e-4097-9e95-fa718424e470-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 00:58:12.455000 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.454963    4888 generic.go:356] "Generic (PLEG): container finished" podID="528ceaae-729e-4097-9e95-fa718424e470" containerID="01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4" exitCode=0
Oct 31 00:58:12.455167 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.455006    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"
Oct 31 00:58:12.455167 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.455056    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" event={"ID":"528ceaae-729e-4097-9e95-fa718424e470","Type":"ContainerDied","Data":"01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4"}
Oct 31 00:58:12.455167 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.455106    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng" event={"ID":"528ceaae-729e-4097-9e95-fa718424e470","Type":"ContainerDied","Data":"20c35be2fe17aa284305d7a2795832ba84c08255a9ee118a429b67c886e0b23b"}
Oct 31 00:58:12.455167 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.455128    4888 scope.go:122] "RemoveContainer" containerID="01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4"
Oct 31 00:58:12.463835 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.463819    4888 scope.go:122] "RemoveContainer" containerID="01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4"
Oct 31 00:58:12.464083 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:58:12.464061    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4\": container with ID starting with 01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4 not found: ID does not exist" containerID="01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4"
Oct 31 00:58:12.464132 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.464091    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4"} err="failed to get container status \"01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4\": rpc error: code = NotFound desc = could not find container \"01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4\": container with ID starting with 01b5c64d137d0f5d507142221a83d36b16c277f772cf577eb343f5851bfc70c4 not found: ID does not exist"
Oct 31 00:58:12.613370 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.613268    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"]
Oct 31 00:58:12.660259 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:12.660226    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-77676c68949nbng"]
Oct 31 00:58:13.553404 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:13.553372    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528ceaae-729e-4097-9e95-fa718424e470" path="/var/lib/kubelet/pods/528ceaae-729e-4097-9e95-fa718424e470/volumes"
Oct 31 00:58:18.551939 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:58:18.551898    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:58:19.327658 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.327618    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"]
Oct 31 00:58:19.328155 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328133    4888 cpu_manager.go:414] "containerMap: removing container" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerName="main"
Oct 31 00:58:19.328155 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328156    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerName="main"
Oct 31 00:58:19.328282 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328182    4888 cpu_manager.go:414] "containerMap: removing container" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main"
Oct 31 00:58:19.328282 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328190    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main"
Oct 31 00:58:19.328282 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328210    4888 cpu_manager.go:414] "containerMap: removing container" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerName="storage-initializer"
Oct 31 00:58:19.328282 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328218    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerName="storage-initializer"
Oct 31 00:58:19.328427 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328311    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65fe27e-a6c2-40b1-9656-0dc6a280123d" containerName="main"
Oct 31 00:58:19.328427 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.328325    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="528ceaae-729e-4097-9e95-fa718424e470" containerName="main"
Oct 31 00:58:19.333068 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.333048    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.380374 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.380322    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"]
Oct 31 00:58:19.401792 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.401762    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tmp-dir\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.401946 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.401795    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz2xr\" (UniqueName: \"kubernetes.io/projected/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kube-api-access-mz2xr\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.401946 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.401826    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kserve-provision-location\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.401946 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.401933    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-model-cache\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.402094 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.401969    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tls-certs\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.402094 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.402001    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f161b12-b4b5-4201-89c7-3e4bccf6135d-cabundle-cert\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.402094 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.402083    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-home\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.402208 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.402106    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-dshm\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.425269 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.425236    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"seaweedfs-s3-creds\""
Oct 31 00:58:19.425269 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.425236    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Oct 31 00:58:19.425269 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.425260    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 00:58:19.425491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.425281    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-6cv2c\""
Oct 31 00:58:19.425491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.425292    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Oct 31 00:58:19.425491 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.425300    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Oct 31 00:58:19.503185 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503159    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tmp-dir\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503193    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz2xr\" (UniqueName: \"kubernetes.io/projected/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kube-api-access-mz2xr\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503212    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kserve-provision-location\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503241    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-model-cache\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503268    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tls-certs\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503291    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f161b12-b4b5-4201-89c7-3e4bccf6135d-cabundle-cert\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503309    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-home\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503343 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503333    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-dshm\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503731 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503668    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kserve-provision-location\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503731 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503707    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tmp-dir\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503814 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503780    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-model-cache\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.503916 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503896    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-home\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.504016 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.503998    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f161b12-b4b5-4201-89c7-3e4bccf6135d-cabundle-cert\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.505782 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.505762    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-dshm\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.505885 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.505865    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tls-certs\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.562911 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.562877    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz2xr\" (UniqueName: \"kubernetes.io/projected/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kube-api-access-mz2xr\") pod \"auth-disabled-test-kserve-5d799fb546-rqml6\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.649137 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.649044    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:19.787781 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.787743    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"]
Oct 31 00:58:19.791281 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.791256    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:19.805709 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.805684    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnnc7\" (UniqueName: \"kubernetes.io/projected/892fc2b5-f429-481e-b657-257fa9f5cb6a-kube-api-access-jnnc7\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:19.805805 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.805718    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892fc2b5-f429-481e-b657-257fa9f5cb6a-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:19.843367 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.843329    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-4k22t\""
Oct 31 00:58:19.849105 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.849081    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"]
Oct 31 00:58:19.906634 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.906568    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnnc7\" (UniqueName: \"kubernetes.io/projected/892fc2b5-f429-481e-b657-257fa9f5cb6a-kube-api-access-jnnc7\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:19.906634 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.906605    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892fc2b5-f429-481e-b657-257fa9f5cb6a-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:19.909050 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.909031    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892fc2b5-f429-481e-b657-257fa9f5cb6a-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:19.968050 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:19.968019    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnnc7\" (UniqueName: \"kubernetes.io/projected/892fc2b5-f429-481e-b657-257fa9f5cb6a-kube-api-access-jnnc7\") pod \"auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:20.103059 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:20.103012    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:20.119802 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:20.119778    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"]
Oct 31 00:58:20.126132 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:58:20.126104    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f161b12_b4b5_4201_89c7_3e4bccf6135d.slice/crio-835a00b5cfb860ffa6863aaa4c28da6cdd845013bc20ec9170c3a70d85221b7d WatchSource:0}: Error finding container 835a00b5cfb860ffa6863aaa4c28da6cdd845013bc20ec9170c3a70d85221b7d: Status 404 returned error can't find the container with id 835a00b5cfb860ffa6863aaa4c28da6cdd845013bc20ec9170c3a70d85221b7d
Oct 31 00:58:20.483814 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:20.483786    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"]
Oct 31 00:58:20.491528 ip-10-0-131-97 kubenswrapper[4888]: W1031 00:58:20.491498    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892fc2b5_f429_481e_b657_257fa9f5cb6a.slice/crio-680fd655e6ad0a65f2cd1bd18850d0248da1ad96f66ac8b38f244429df7fff44 WatchSource:0}: Error finding container 680fd655e6ad0a65f2cd1bd18850d0248da1ad96f66ac8b38f244429df7fff44: Status 404 returned error can't find the container with id 680fd655e6ad0a65f2cd1bd18850d0248da1ad96f66ac8b38f244429df7fff44
Oct 31 00:58:20.496095 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:20.496066    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" event={"ID":"9f161b12-b4b5-4201-89c7-3e4bccf6135d","Type":"ContainerStarted","Data":"a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a"}
Oct 31 00:58:20.496189 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:20.496116    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" event={"ID":"9f161b12-b4b5-4201-89c7-3e4bccf6135d","Type":"ContainerStarted","Data":"835a00b5cfb860ffa6863aaa4c28da6cdd845013bc20ec9170c3a70d85221b7d"}
Oct 31 00:58:21.503257 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:21.503212    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" event={"ID":"892fc2b5-f429-481e-b657-257fa9f5cb6a","Type":"ContainerStarted","Data":"6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7"}
Oct 31 00:58:21.503690 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:21.503303    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:21.503690 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:21.503328    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" event={"ID":"892fc2b5-f429-481e-b657-257fa9f5cb6a","Type":"ContainerStarted","Data":"680fd655e6ad0a65f2cd1bd18850d0248da1ad96f66ac8b38f244429df7fff44"}
Oct 31 00:58:21.617280 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:21.617221    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" podStartSLOduration=2.617203678 podStartE2EDuration="2.617203678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:21.614924542 +0000 UTC m=+1710.169811982" watchObservedRunningTime="2026-10-31 00:58:21.617203678 +0000 UTC m=+1710.172091117"
Oct 31 00:58:22.508748 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:22.508718    4888 generic.go:356] "Generic (PLEG): container finished" podID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerID="a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a" exitCode=0
Oct 31 00:58:22.509132 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:22.508791    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" event={"ID":"9f161b12-b4b5-4201-89c7-3e4bccf6135d","Type":"ContainerDied","Data":"a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a"}
Oct 31 00:58:23.515034 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:23.515004    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" event={"ID":"9f161b12-b4b5-4201-89c7-3e4bccf6135d","Type":"ContainerStarted","Data":"213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19"}
Oct 31 00:58:23.633091 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:23.633041    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podStartSLOduration=2.5525991279999998 podStartE2EDuration="4.633029231s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.080430103s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:58:23.632557677 +0000 UTC m=+1712.187445150" watchObservedRunningTime="2026-10-31 00:58:23.633029231 +0000 UTC m=+1712.187916668"
Oct 31 00:58:23.649572 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:23.649545    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:29.649626 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:29.649587    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 00:58:29.650868 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:29.650845    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:58:39.650549 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:39.650503    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:58:48.552280 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:58:48.552241    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:58:49.649586 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:49.649507    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:58:52.510667 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:52.510640    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 00:58:59.649540 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:58:59.649498    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:59:09.649456 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:09.649400    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:59:18.552025 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:59:18.551982    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:59:19.649722 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:19.649668    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:59:29.650183 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:29.650132    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:59:39.650510 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:39.650464    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:59:48.551496 ip-10-0-131-97 kubenswrapper[4888]: E1031 00:59:48.551460    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 00:59:49.649627 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:49.649547    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 00:59:51.738498 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:51.738466    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:59:51.743531 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:51.743507    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 00:59:59.649762 ip-10-0-131-97 kubenswrapper[4888]: I1031 00:59:59.649713    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Oct 31 01:00:09.659272 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:09.659244    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 01:00:09.666718 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:09.666687    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 01:00:18.554691 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:00:18.554646    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:00:20.971977 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:20.971900    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"]
Oct 31 01:00:21.007931 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:21.007899    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"]
Oct 31 01:00:22.510639 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:22.510599    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:00:32.510786 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:32.510733    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:00:35.991711 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:35.991674    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main" containerID="cri-o://6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7" gracePeriod=45
Oct 31 01:00:36.031385 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.031319    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" containerID="cri-o://213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19" gracePeriod=45
Oct 31 01:00:36.666146 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.666124    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 01:00:36.742707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742680    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tmp-dir\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.742875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742724    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-home\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-home\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.742875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742749    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kserve-provision-location\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.742875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742783    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-model-cache\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.742875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742822    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kube-api-access-mz2xr\" (UniqueName: \"kubernetes.io/projected/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kube-api-access-mz2xr\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.742875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742840    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f161b12-b4b5-4201-89c7-3e4bccf6135d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f161b12-b4b5-4201-89c7-3e4bccf6135d-cabundle-cert\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.743142 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742877    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-dshm\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.743142 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.742910    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tls-certs\") pod \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\" (UID: \"9f161b12-b4b5-4201-89c7-3e4bccf6135d\") "
Oct 31 01:00:36.743142 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.743092    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-model-cache" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:36.743482 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.743457    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-home" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:36.743600 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.743558    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f161b12-b4b5-4201-89c7-3e4bccf6135d-cabundle-cert" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:36.745303 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.745280    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tls-certs" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:00:36.745405 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.745280    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kube-api-access-mz2xr" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "kube-api-access-mz2xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:36.745473 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.745455    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-dshm" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:36.755260 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.755238    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tmp-dir" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:36.764122 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.764069    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kserve-provision-location" pod "9f161b12-b4b5-4201-89c7-3e4bccf6135d" (UID: "9f161b12-b4b5-4201-89c7-3e4bccf6135d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:36.843458 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843434    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:36.843458 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843456    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:36.843624 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843467    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:36.843624 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843478    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz2xr\" (UniqueName: \"kubernetes.io/projected/9f161b12-b4b5-4201-89c7-3e4bccf6135d-kube-api-access-mz2xr\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:36.843624 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843493    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f161b12-b4b5-4201-89c7-3e4bccf6135d-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:36.843624 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843501    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:36.843624 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843509    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:36.843624 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:36.843516    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f161b12-b4b5-4201-89c7-3e4bccf6135d-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.151842 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.151757    4888 generic.go:356] "Generic (PLEG): container finished" podID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerID="213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19" exitCode=0
Oct 31 01:00:37.151842 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.151814    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"
Oct 31 01:00:37.152317 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.151850    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" event={"ID":"9f161b12-b4b5-4201-89c7-3e4bccf6135d","Type":"ContainerDied","Data":"213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19"}
Oct 31 01:00:37.152317 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.151908    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" event={"ID":"9f161b12-b4b5-4201-89c7-3e4bccf6135d","Type":"ContainerDied","Data":"835a00b5cfb860ffa6863aaa4c28da6cdd845013bc20ec9170c3a70d85221b7d"}
Oct 31 01:00:37.152317 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.151938    4888 scope.go:122] "RemoveContainer" containerID="213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19"
Oct 31 01:00:37.162563 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.162539    4888 scope.go:122] "RemoveContainer" containerID="a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a"
Oct 31 01:00:37.170367 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.170325    4888 scope.go:122] "RemoveContainer" containerID="213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19"
Oct 31 01:00:37.170622 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:00:37.170605    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19\": container with ID starting with 213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19 not found: ID does not exist" containerID="213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19"
Oct 31 01:00:37.170664 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.170631    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19"} err="failed to get container status \"213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19\": rpc error: code = NotFound desc = could not find container \"213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19\": container with ID starting with 213d5f2eb5f26c38e2c5cdca5c462b9e5ece1e3a4508a11945f9ea4fe8687f19 not found: ID does not exist"
Oct 31 01:00:37.170664 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.170647    4888 scope.go:122] "RemoveContainer" containerID="a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a"
Oct 31 01:00:37.170894 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:00:37.170872    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a\": container with ID starting with a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a not found: ID does not exist" containerID="a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a"
Oct 31 01:00:37.170981 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.170900    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a"} err="failed to get container status \"a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a\": rpc error: code = NotFound desc = could not find container \"a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a\": container with ID starting with a0041efee06d0958f9d1419f0f498e58cae2691e6e1a86536b4245db4188d30a not found: ID does not exist"
Oct 31 01:00:37.312968 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.312936    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"]
Oct 31 01:00:37.319091 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.319065    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6"]
Oct 31 01:00:37.553036 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.553001    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" path="/var/lib/kubelet/pods/9f161b12-b4b5-4201-89c7-3e4bccf6135d/volumes"
Oct 31 01:00:37.660648 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:37.660608    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-5d799fb546-rqml6" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Oct 31 01:00:42.510779 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:42.510734    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:00:42.511268 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:42.510872    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 01:00:48.551428 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:00:48.551391    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:00:52.510638 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:00:52.510603    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:02.510431 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:02.510397    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:01:06.237433 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.237408    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 01:01:06.290879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.290807    4888 generic.go:356] "Generic (PLEG): container finished" podID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerID="6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7" exitCode=0
Oct 31 01:01:06.290879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.290856    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"
Oct 31 01:01:06.291047 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.290889    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" event={"ID":"892fc2b5-f429-481e-b657-257fa9f5cb6a","Type":"ContainerDied","Data":"6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7"}
Oct 31 01:01:06.291047 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.290940    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr" event={"ID":"892fc2b5-f429-481e-b657-257fa9f5cb6a","Type":"ContainerDied","Data":"680fd655e6ad0a65f2cd1bd18850d0248da1ad96f66ac8b38f244429df7fff44"}
Oct 31 01:01:06.291047 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.290965    4888 scope.go:122] "RemoveContainer" containerID="6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7"
Oct 31 01:01:06.298694 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.298678    4888 scope.go:122] "RemoveContainer" containerID="6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7"
Oct 31 01:01:06.298933 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:01:06.298917    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7\": container with ID starting with 6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7 not found: ID does not exist" containerID="6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7"
Oct 31 01:01:06.298972 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.298943    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7"} err="failed to get container status \"6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7\": rpc error: code = NotFound desc = could not find container \"6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7\": container with ID starting with 6892ef55b3e668d46cfbd5a30914cc2e68273d0c7451b4f07db07946a7e536f7 not found: ID does not exist"
Oct 31 01:01:06.381172 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.381144    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892fc2b5-f429-481e-b657-257fa9f5cb6a-kube-api-access-jnnc7\" (UniqueName: \"kubernetes.io/projected/892fc2b5-f429-481e-b657-257fa9f5cb6a-kube-api-access-jnnc7\") pod \"892fc2b5-f429-481e-b657-257fa9f5cb6a\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") "
Oct 31 01:01:06.381329 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.381203    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/892fc2b5-f429-481e-b657-257fa9f5cb6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/892fc2b5-f429-481e-b657-257fa9f5cb6a-tls-certs\") pod \"892fc2b5-f429-481e-b657-257fa9f5cb6a\" (UID: \"892fc2b5-f429-481e-b657-257fa9f5cb6a\") "
Oct 31 01:01:06.383406 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.383373    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/892fc2b5-f429-481e-b657-257fa9f5cb6a-tls-certs" pod "892fc2b5-f429-481e-b657-257fa9f5cb6a" (UID: "892fc2b5-f429-481e-b657-257fa9f5cb6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:01:06.383529 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.383443    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892fc2b5-f429-481e-b657-257fa9f5cb6a-kube-api-access-jnnc7" pod "892fc2b5-f429-481e-b657-257fa9f5cb6a" (UID: "892fc2b5-f429-481e-b657-257fa9f5cb6a"). InnerVolumeSpecName "kube-api-access-jnnc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:06.482407 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.482381    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnnc7\" (UniqueName: \"kubernetes.io/projected/892fc2b5-f429-481e-b657-257fa9f5cb6a-kube-api-access-jnnc7\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:01:06.482407 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.482404    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/892fc2b5-f429-481e-b657-257fa9f5cb6a-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:01:06.706884 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.706856    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"]
Oct 31 01:01:06.713427 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:06.713399    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-5d987cbbfd-qqfpr"]
Oct 31 01:01:07.553794 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:07.553761    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" path="/var/lib/kubelet/pods/892fc2b5-f429-481e-b657-257fa9f5cb6a/volumes"
Oct 31 01:01:17.501957 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.501927    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"]
Oct 31 01:01:17.502334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502222    4888 cpu_manager.go:414] "containerMap: removing container" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="storage-initializer"
Oct 31 01:01:17.502334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502232    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="storage-initializer"
Oct 31 01:01:17.502334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502252    4888 cpu_manager.go:414] "containerMap: removing container" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main"
Oct 31 01:01:17.502334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502257    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main"
Oct 31 01:01:17.502334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502278    4888 cpu_manager.go:414] "containerMap: removing container" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main"
Oct 31 01:01:17.502334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502283    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main"
Oct 31 01:01:17.502334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502329    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f161b12-b4b5-4201-89c7-3e4bccf6135d" containerName="main"
Oct 31 01:01:17.502566 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.502340    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="892fc2b5-f429-481e-b657-257fa9f5cb6a" containerName="main"
Oct 31 01:01:17.504844 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.504823    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.602291 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.602258    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Oct 31 01:01:17.602291 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.602268    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Oct 31 01:01:17.602496 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.602300    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-dnrcc\""
Oct 31 01:01:17.602496 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.602333    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Oct 31 01:01:17.604317 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.604300    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"]
Oct 31 01:01:17.674780 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.674738    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbh28\" (UniqueName: \"kubernetes.io/projected/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-kube-api-access-sbh28\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.674954 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.674786    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.674954 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.674890    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.674954 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.674919    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.675090 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.674976    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.675148 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.675114    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776267 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776186    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776267 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776235    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776267 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776263    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbh28\" (UniqueName: \"kubernetes.io/projected/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-kube-api-access-sbh28\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776569 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776282    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776569 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776307    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776569 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776496    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776705 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776632    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776771 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776702    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.776771 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.776741    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.778605 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.778584    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.778874 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.778858    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:17.836521 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:17.836485    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbh28\" (UniqueName: \"kubernetes.io/projected/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-kube-api-access-sbh28\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:18.115872 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:18.115789    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:18.487303 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:18.487267    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"]
Oct 31 01:01:18.492455 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:01:18.492429    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1c4a7b_c7e8_42db_8dcb_e83b83646b6e.slice/crio-c1c854e19762b7069f5699607aaa29d5adb85d761011acd714fcf36c7cfc97aa WatchSource:0}: Error finding container c1c854e19762b7069f5699607aaa29d5adb85d761011acd714fcf36c7cfc97aa: Status 404 returned error can't find the container with id c1c854e19762b7069f5699607aaa29d5adb85d761011acd714fcf36c7cfc97aa
Oct 31 01:01:18.552308 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:01:18.552280    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:01:19.356126 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:19.356085    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" event={"ID":"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e","Type":"ContainerStarted","Data":"e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a"}
Oct 31 01:01:19.356334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:19.356143    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" event={"ID":"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e","Type":"ContainerStarted","Data":"c1c854e19762b7069f5699607aaa29d5adb85d761011acd714fcf36c7cfc97aa"}
Oct 31 01:01:19.469685 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:19.469638    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" podStartSLOduration=2.469623813 podStartE2EDuration="2.469623813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:19.468953571 +0000 UTC m=+1888.023841009" watchObservedRunningTime="2026-10-31 01:01:19.469623813 +0000 UTC m=+1888.024511250"
Oct 31 01:01:20.115911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:20.115871    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:28.116532 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:28.116494    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:28.128497 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:28.128472    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:28.407813 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:01:28.407737    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:01:48.551841 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:01:48.551804    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:02:18.551783 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:02:18.551745    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:02:32.772227 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:32.772197    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"]
Oct 31 01:02:32.781561 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:02:32.781525    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a" cmd=["/bin/sleep","15"]
Oct 31 01:02:32.781707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:32.781589    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 01:02:32.781707 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:02:32.781604    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" podUID="ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" containerName="main" containerID="cri-o://e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a"
Oct 31 01:02:32.781707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:32.781653    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" podUID="ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" containerName="main" containerID="cri-o://e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a" gracePeriod=60
Oct 31 01:02:33.129069 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.129045    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:02:33.254828 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.254800    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-home\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-home\") pod \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") "
Oct 31 01:02:33.255002 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.254861    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-dshm\") pod \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") "
Oct 31 01:02:33.255002 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.254888    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-kube-api-access-sbh28\" (UniqueName: \"kubernetes.io/projected/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-kube-api-access-sbh28\") pod \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") "
Oct 31 01:02:33.255002 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.254903    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tmp-dir\") pod \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") "
Oct 31 01:02:33.255002 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.254951    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tls-certs\") pod \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") "
Oct 31 01:02:33.255002 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.254974    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-model-cache\") pod \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\" (UID: \"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e\") "
Oct 31 01:02:33.255341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.255097    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-home" pod "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" (UID: "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:33.255341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.255187    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tmp-dir" pod "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" (UID: "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:33.255341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.255209    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:33.255341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.255289    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-model-cache" pod "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" (UID: "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:33.257224 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.257195    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-dshm" pod "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" (UID: "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:02:33.257590 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.257563    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-kube-api-access-sbh28" pod "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" (UID: "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e"). InnerVolumeSpecName "kube-api-access-sbh28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:02:33.258436 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.258414    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tls-certs" pod "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" (UID: "ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:02:33.356128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.356064    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:33.356128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.356090    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:33.356128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.356100    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:33.356128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.356110    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbh28\" (UniqueName: \"kubernetes.io/projected/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-kube-api-access-sbh28\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:33.356128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.356119    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:02:33.682693 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.682619    4888 generic.go:356] "Generic (PLEG): container finished" podID="ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" containerID="e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a" exitCode=0
Oct 31 01:02:33.682693 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.682663    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"
Oct 31 01:02:33.682936 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.682707    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" event={"ID":"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e","Type":"ContainerDied","Data":"e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a"}
Oct 31 01:02:33.682936 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.682766    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr" event={"ID":"ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e","Type":"ContainerDied","Data":"c1c854e19762b7069f5699607aaa29d5adb85d761011acd714fcf36c7cfc97aa"}
Oct 31 01:02:33.682936 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.682804    4888 scope.go:122] "RemoveContainer" containerID="e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a"
Oct 31 01:02:33.691400 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.691381    4888 scope.go:122] "RemoveContainer" containerID="e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a"
Oct 31 01:02:33.691636 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:02:33.691620    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a\": container with ID starting with e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a not found: ID does not exist" containerID="e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a"
Oct 31 01:02:33.691685 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.691646    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a"} err="failed to get container status \"e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a\": rpc error: code = NotFound desc = could not find container \"e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a\": container with ID starting with e08e837a7c2ec7942b078cae970fe18d1d319f8c062a9a4d03eed6b954f6767a not found: ID does not exist"
Oct 31 01:02:33.841525 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.841494    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"]
Oct 31 01:02:33.847892 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:33.847868    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5577b665bb49mnr"]
Oct 31 01:02:35.553592 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:35.553561    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" path="/var/lib/kubelet/pods/ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e/volumes"
Oct 31 01:02:45.109191 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.109152    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"]
Oct 31 01:02:45.109723 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.109529    4888 cpu_manager.go:414] "containerMap: removing container" podUID="ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" containerName="main"
Oct 31 01:02:45.109723 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.109541    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" containerName="main"
Oct 31 01:02:45.109723 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.109593    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac1c4a7b-c7e8-42db-8dcb-e83b83646b6e" containerName="main"
Oct 31 01:02:45.114940 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.114910    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.170146 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.170118    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"]
Oct 31 01:02:45.215075 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.215031    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-m5prz\""
Oct 31 01:02:45.215075 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.215035    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Oct 31 01:02:45.215260 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.215097    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-fck64\""
Oct 31 01:02:45.215573 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.215556    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Oct 31 01:02:45.215645 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.215627    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Oct 31 01:02:45.250492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.250464    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.250492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.250496    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwvln\" (UniqueName: \"kubernetes.io/projected/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-kube-api-access-kwvln\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.351416 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.351383    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.351621 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.351426    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwvln\" (UniqueName: \"kubernetes.io/projected/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-kube-api-access-kwvln\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.354000 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.353976    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.412828 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.412756    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwvln\" (UniqueName: \"kubernetes.io/projected/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-kube-api-access-kwvln\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.427315 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.427292    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:45.798615 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:45.798584    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"]
Oct 31 01:02:45.806431 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:02:45.806403    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb5c4b9_0ce3_44a3_9261_c6bc66d9aa6a.slice/crio-e0d5429f34b0bfed687574c7ad4fdaa7e8741e9df0d1f43f149e0c26edeffe85 WatchSource:0}: Error finding container e0d5429f34b0bfed687574c7ad4fdaa7e8741e9df0d1f43f149e0c26edeffe85: Status 404 returned error can't find the container with id e0d5429f34b0bfed687574c7ad4fdaa7e8741e9df0d1f43f149e0c26edeffe85
Oct 31 01:02:46.744775 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:46.744733    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" event={"ID":"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a","Type":"ContainerStarted","Data":"d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405"}
Oct 31 01:02:46.745299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:46.744821    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" event={"ID":"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a","Type":"ContainerStarted","Data":"e0d5429f34b0bfed687574c7ad4fdaa7e8741e9df0d1f43f149e0c26edeffe85"}
Oct 31 01:02:46.745299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:46.744852    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:02:46.858423 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:02:46.858375    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" podStartSLOduration=1.858336325 podStartE2EDuration="1.858336325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:02:46.857459175 +0000 UTC m=+1975.412346612" watchObservedRunningTime="2026-10-31 01:02:46.858336325 +0000 UTC m=+1975.413223762"
Oct 31 01:02:48.566581 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:02:48.566547    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:03:17.751116 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:17.751085    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:03:18.552002 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:03:18.551969    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:03:25.983506 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:25.983469    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"]
Oct 31 01:03:25.987141 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:25.987120    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.030313 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.030284    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"]
Oct 31 01:03:26.033452 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.033437    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.080900 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.080873    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081002 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.080908    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081002 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.080933    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081086 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.081051    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14c415a-7a4a-402e-b3f4-f5531312db7e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081122 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.081094    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081122 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.081111    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwfq4\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-kube-api-access-hwfq4\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081195 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.081142    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081195 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.081160    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.081266 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.081238    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.084947 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.084927    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"]
Oct 31 01:03:26.133483 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.133458    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"]
Oct 31 01:03:26.182092 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182062    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182239 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182101    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.182239 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182122    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.182239 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182149    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182239 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182222    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14c415a-7a4a-402e-b3f4-f5531312db7e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182272    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.182492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182304    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b5765b-2435-4523-a9cb-8a60354078ec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.182492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182329    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182439    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwfq4\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-kube-api-access-hwfq4\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182473    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.182750 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182515    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcj5\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-kube-api-access-phcj5\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.182750 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182543    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.182750 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182572    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182750 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182640    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182750 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182679    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.182750 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182740    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.183042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182759    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.183042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182772    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.183042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182787    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.183042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182868    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.183042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.182999    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.183042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.183021    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14c415a-7a4a-402e-b3f4-f5531312db7e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.183042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.183005    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.184826 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.184802    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.185371 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.185327    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.242409 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.242320    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.242523 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.242490    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwfq4\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-kube-api-access-hwfq4\") pod \"openshift-ai-inference-openshift-default-568c495856-dcvp9\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.284090 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284067    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284218 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284096    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284218 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284124    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284320 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284233    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284320 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284284    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284320 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284313    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b5765b-2435-4523-a9cb-8a60354078ec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284497 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284342    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284497 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284409    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcj5\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-kube-api-access-phcj5\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284497 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284437    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284497 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284460    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284694 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284549    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284694 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284657    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.284818 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.284797    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.285247 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.285226    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b5765b-2435-4523-a9cb-8a60354078ec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.286232 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.286217    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.286730 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.286707    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.299640 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.299619    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:26.344017 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.343985    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.344303 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.344279    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcj5\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-kube-api-access-phcj5\") pod \"openshift-ai-inference-openshift-default-568c495856-c7ln8\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.344473 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.344457    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:26.725075 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.725041    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"]
Oct 31 01:03:26.732793 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:03:26.732769    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14c415a_7a4a_402e_b3f4_f5531312db7e.slice/crio-981bb20459413ade34be06c88e16e3320d64fadd6707d78fd481afd1a985a33d WatchSource:0}: Error finding container 981bb20459413ade34be06c88e16e3320d64fadd6707d78fd481afd1a985a33d: Status 404 returned error can't find the container with id 981bb20459413ade34be06c88e16e3320d64fadd6707d78fd481afd1a985a33d
Oct 31 01:03:26.734702 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.734669    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:03:26.734781 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.734736    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:03:26.734781 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.734768    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:03:26.781781 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.781760    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"]
Oct 31 01:03:26.789056 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:03:26.789021    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b5765b_2435_4523_a9cb_8a60354078ec.slice/crio-8e103f1b8373f8f3e5fe4fbdf66429a97b159a79c5ba4f0e7317df3b4664e0eb WatchSource:0}: Error finding container 8e103f1b8373f8f3e5fe4fbdf66429a97b159a79c5ba4f0e7317df3b4664e0eb: Status 404 returned error can't find the container with id 8e103f1b8373f8f3e5fe4fbdf66429a97b159a79c5ba4f0e7317df3b4664e0eb
Oct 31 01:03:26.790963 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.790932    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:03:26.791070 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.790995    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:03:26.791070 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.791023    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:03:26.926414 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.926373    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" event={"ID":"f14c415a-7a4a-402e-b3f4-f5531312db7e","Type":"ContainerStarted","Data":"02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42"}
Oct 31 01:03:26.926597 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.926443    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" event={"ID":"f14c415a-7a4a-402e-b3f4-f5531312db7e","Type":"ContainerStarted","Data":"981bb20459413ade34be06c88e16e3320d64fadd6707d78fd481afd1a985a33d"}
Oct 31 01:03:26.928173 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.928146    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" event={"ID":"b6b5765b-2435-4523-a9cb-8a60354078ec","Type":"ContainerStarted","Data":"ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba"}
Oct 31 01:03:26.928294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:26.928196    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" event={"ID":"b6b5765b-2435-4523-a9cb-8a60354078ec","Type":"ContainerStarted","Data":"8e103f1b8373f8f3e5fe4fbdf66429a97b159a79c5ba4f0e7317df3b4664e0eb"}
Oct 31 01:03:27.043022 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.042922    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podStartSLOduration=2.042904348 podStartE2EDuration="2.042904348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:27.042678361 +0000 UTC m=+2015.597565804" watchObservedRunningTime="2026-10-31 01:03:27.042904348 +0000 UTC m=+2015.597791787"
Oct 31 01:03:27.159217 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.159051    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podStartSLOduration=2.159032811 podStartE2EDuration="2.159032811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:27.158506412 +0000 UTC m=+2015.713393850" watchObservedRunningTime="2026-10-31 01:03:27.159032811 +0000 UTC m=+2015.713920249"
Oct 31 01:03:27.299837 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.299760    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:27.300855 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.300836    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-dcvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:03:27.300911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.300880    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:03:27.350371 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.350013    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:27.350371 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.350146    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-c7ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:03:27.350371 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:27.350191    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 01:03:28.300957 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:28.300925    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-dcvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:03:28.301376 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:28.300978    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:03:28.348847 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:28.348722    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-c7ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:03:28.348847 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:28.348787    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 01:03:29.300684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:29.300647    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-dcvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:03:29.300869 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:29.300699    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:03:29.348374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:29.344892    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-c7ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:03:29.348374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:29.344955    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 01:03:30.300690 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:30.300649    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-dcvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:03:30.300897 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:30.300720    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:03:30.351373 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:30.349721    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-c7ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:03:30.351373 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:30.349786    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 01:03:31.300825 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:31.300789    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-dcvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:03:31.301021 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:31.300843    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:03:31.348764 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:31.348722    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-c7ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:03:31.349036 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:31.349009    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 01:03:32.300716 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:32.300682    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-dcvp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 31 01:03:32.301110 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:32.300746    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 31 01:03:32.353844 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:32.353721    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-c7ln8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 31 01:03:32.353844 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:32.353784    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 31 01:03:33.303571 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:33.303541    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:33.303960 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:33.303872    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:33.304536 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:33.304519    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:03:33.348949 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:33.348921    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:33.349196 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:33.349180    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:33.349837 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:33.349821    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:03:38.753560 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.753520    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"]
Oct 31 01:03:38.756813 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.756792    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.830494 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.830461    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.830663 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.830504    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.830663 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.830530    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a198573c-1780-422c-a752-279369b14bc7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.830663 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.830594    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8f8j\" (UniqueName: \"kubernetes.io/projected/a198573c-1780-422c-a752-279369b14bc7-kube-api-access-k8f8j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.830783 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.830686    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.830846 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.830825    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.854128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.854098    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Oct 31 01:03:38.854128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.854104    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Oct 31 01:03:38.854299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.854149    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-tcfbx\""
Oct 31 01:03:38.854299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.854205    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Oct 31 01:03:38.856933 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.856914    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"]
Oct 31 01:03:38.931463 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931429    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.931636 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931501    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.931636 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931525    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.931636 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931562    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.931774 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931675    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a198573c-1780-422c-a752-279369b14bc7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.931774 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931730    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8f8j\" (UniqueName: \"kubernetes.io/projected/a198573c-1780-422c-a752-279369b14bc7-kube-api-access-k8f8j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.931964 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931941    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.931964 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.931955    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.932057 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.932039    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.933773 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.933751    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:38.934110 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:38.934095    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a198573c-1780-422c-a752-279369b14bc7-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:39.003858 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.003761    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8f8j\" (UniqueName: \"kubernetes.io/projected/a198573c-1780-422c-a752-279369b14bc7-kube-api-access-k8f8j\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:39.069914 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.069874    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:39.265485 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.265402    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"]
Oct 31 01:03:39.268919 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.268890    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.320504 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.320472    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-zk2x4\""
Oct 31 01:03:39.326130 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.326100    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"]
Oct 31 01:03:39.335251 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.335225    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ckd2\" (UniqueName: \"kubernetes.io/projected/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-kube-api-access-5ckd2\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.335443 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.335282    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.448381 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.439391    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ckd2\" (UniqueName: \"kubernetes.io/projected/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-kube-api-access-5ckd2\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.448381 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.439489    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.448381 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.447413    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"]
Oct 31 01:03:39.448381 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.448291    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.454849 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:03:39.454821    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda198573c_1780_422c_a752_279369b14bc7.slice/crio-0a3adb1a4987f513adb40ca070cb42c1542242085174156af3e62a9e51c77d18 WatchSource:0}: Error finding container 0a3adb1a4987f513adb40ca070cb42c1542242085174156af3e62a9e51c77d18: Status 404 returned error can't find the container with id 0a3adb1a4987f513adb40ca070cb42c1542242085174156af3e62a9e51c77d18
Oct 31 01:03:39.500250 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.500223    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ckd2\" (UniqueName: \"kubernetes.io/projected/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-kube-api-access-5ckd2\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.581565 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.581489    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:39.960019 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:39.959991    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"]
Oct 31 01:03:39.967752 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:03:39.967728    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed6435e_dd31_4b1d_a92c_d7838f6b280b.slice/crio-5ed400cea708be5209aa7fbc6932f61a682c008bb6981fb330b3ffc8494660f7 WatchSource:0}: Error finding container 5ed400cea708be5209aa7fbc6932f61a682c008bb6981fb330b3ffc8494660f7: Status 404 returned error can't find the container with id 5ed400cea708be5209aa7fbc6932f61a682c008bb6981fb330b3ffc8494660f7
Oct 31 01:03:40.028606 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:40.028574    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" event={"ID":"a198573c-1780-422c-a752-279369b14bc7","Type":"ContainerStarted","Data":"5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb"}
Oct 31 01:03:40.028717 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:40.028640    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" event={"ID":"a198573c-1780-422c-a752-279369b14bc7","Type":"ContainerStarted","Data":"0a3adb1a4987f513adb40ca070cb42c1542242085174156af3e62a9e51c77d18"}
Oct 31 01:03:40.030081 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:40.030059    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" event={"ID":"7ed6435e-dd31-4b1d-a92c-d7838f6b280b","Type":"ContainerStarted","Data":"5ed400cea708be5209aa7fbc6932f61a682c008bb6981fb330b3ffc8494660f7"}
Oct 31 01:03:40.070533 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:40.070503    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:40.143162 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:40.143108    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" podStartSLOduration=2.143093519 podStartE2EDuration="2.143093519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:40.141669364 +0000 UTC m=+2028.696556801" watchObservedRunningTime="2026-10-31 01:03:40.143093519 +0000 UTC m=+2028.697980956"
Oct 31 01:03:41.037028 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:41.036981    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" event={"ID":"7ed6435e-dd31-4b1d-a92c-d7838f6b280b","Type":"ContainerStarted","Data":"1eb6e312303af46c6148f8d75082049119341a7297b369a98fcc70feeeb1b8ac"}
Oct 31 01:03:41.037515 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:41.037122    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:03:41.149994 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:41.149947    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" podStartSLOduration=2.149932864 podStartE2EDuration="2.149932864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:41.148926095 +0000 UTC m=+2029.703813534" watchObservedRunningTime="2026-10-31 01:03:41.149932864 +0000 UTC m=+2029.704820301"
Oct 31 01:03:48.551180 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:03:48.551140    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:03:49.070970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:49.070933    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:49.083003 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:49.082972    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:03:49.093688 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:03:49.093662    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:04:12.043458 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:12.043428    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:04:18.552284 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:04:18.552249    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:04:19.228981 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.228934    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"]
Oct 31 01:04:19.231226 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.231188    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"]
Oct 31 01:04:19.239531 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:04:19.239494    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb" cmd=["/bin/sleep","15"]
Oct 31 01:04:19.239653 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.239553    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 01:04:19.239653 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:04:19.239572    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" podUID="a198573c-1780-422c-a752-279369b14bc7" containerName="main" containerID="cri-o://5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb"
Oct 31 01:04:19.239653 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.239619    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" podUID="a198573c-1780-422c-a752-279369b14bc7" containerName="main" containerID="cri-o://5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb" gracePeriod=60
Oct 31 01:04:19.593376 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.593334    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:04:19.685532 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685498    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a198573c-1780-422c-a752-279369b14bc7-kube-api-access-k8f8j\" (UniqueName: \"kubernetes.io/projected/a198573c-1780-422c-a752-279369b14bc7-kube-api-access-k8f8j\") pod \"a198573c-1780-422c-a752-279369b14bc7\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") "
Oct 31 01:04:19.685812 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685607    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-tmp-dir\") pod \"a198573c-1780-422c-a752-279369b14bc7\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") "
Oct 31 01:04:19.685812 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685632    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-model-cache\") pod \"a198573c-1780-422c-a752-279369b14bc7\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") "
Oct 31 01:04:19.685812 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685648    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-dshm\") pod \"a198573c-1780-422c-a752-279369b14bc7\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") "
Oct 31 01:04:19.685812 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685695    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-home\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-home\") pod \"a198573c-1780-422c-a752-279369b14bc7\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") "
Oct 31 01:04:19.685812 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685724    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a198573c-1780-422c-a752-279369b14bc7-tls-certs\" (UniqueName: \"kubernetes.io/secret/a198573c-1780-422c-a752-279369b14bc7-tls-certs\") pod \"a198573c-1780-422c-a752-279369b14bc7\" (UID: \"a198573c-1780-422c-a752-279369b14bc7\") "
Oct 31 01:04:19.686051 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685885    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-tmp-dir" pod "a198573c-1780-422c-a752-279369b14bc7" (UID: "a198573c-1780-422c-a752-279369b14bc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:19.686051 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.685902    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-model-cache" pod "a198573c-1780-422c-a752-279369b14bc7" (UID: "a198573c-1780-422c-a752-279369b14bc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:19.686207 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.686186    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-home" pod "a198573c-1780-422c-a752-279369b14bc7" (UID: "a198573c-1780-422c-a752-279369b14bc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:19.687945 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.687909    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-dshm" pod "a198573c-1780-422c-a752-279369b14bc7" (UID: "a198573c-1780-422c-a752-279369b14bc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:04:19.687945 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.687932    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a198573c-1780-422c-a752-279369b14bc7-kube-api-access-k8f8j" pod "a198573c-1780-422c-a752-279369b14bc7" (UID: "a198573c-1780-422c-a752-279369b14bc7"). InnerVolumeSpecName "kube-api-access-k8f8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:04:19.688072 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.687938    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a198573c-1780-422c-a752-279369b14bc7-tls-certs" pod "a198573c-1780-422c-a752-279369b14bc7" (UID: "a198573c-1780-422c-a752-279369b14bc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:04:19.786543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.786454    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:19.786543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.786486    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:19.786543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.786497    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:19.786543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.786506    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a198573c-1780-422c-a752-279369b14bc7-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:19.786543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.786513    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a198573c-1780-422c-a752-279369b14bc7-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:19.786543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:19.786522    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8f8j\" (UniqueName: \"kubernetes.io/projected/a198573c-1780-422c-a752-279369b14bc7-kube-api-access-k8f8j\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:04:20.229489 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.229449    4888 generic.go:356] "Generic (PLEG): container finished" podID="a198573c-1780-422c-a752-279369b14bc7" containerID="5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb" exitCode=0
Oct 31 01:04:20.229659 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.229508    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"
Oct 31 01:04:20.229659 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.229544    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" event={"ID":"a198573c-1780-422c-a752-279369b14bc7","Type":"ContainerDied","Data":"5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb"}
Oct 31 01:04:20.229659 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.229597    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl" event={"ID":"a198573c-1780-422c-a752-279369b14bc7","Type":"ContainerDied","Data":"0a3adb1a4987f513adb40ca070cb42c1542242085174156af3e62a9e51c77d18"}
Oct 31 01:04:20.229659 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.229618    4888 scope.go:122] "RemoveContainer" containerID="5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb"
Oct 31 01:04:20.238863 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.238844    4888 scope.go:122] "RemoveContainer" containerID="5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb"
Oct 31 01:04:20.239145 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:04:20.239125    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb\": container with ID starting with 5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb not found: ID does not exist" containerID="5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb"
Oct 31 01:04:20.239207 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.239160    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb"} err="failed to get container status \"5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb\": rpc error: code = NotFound desc = could not find container \"5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb\": container with ID starting with 5b7980f93bd7afe901663ec1242a524cf0c5b1122cc34b348f679f7180a6a6fb not found: ID does not exist"
Oct 31 01:04:20.388777 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.388741    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"]
Oct 31 01:04:20.435333 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:20.435302    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5f76f74777hfskl"]
Oct 31 01:04:21.553802 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:21.553773    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a198573c-1780-422c-a752-279369b14bc7" path="/var/lib/kubelet/pods/a198573c-1780-422c-a752-279369b14bc7/volumes"
Oct 31 01:04:22.042552 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:22.042510    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:32.043177 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:32.043137    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:34.252643 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:34.252580    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main" containerID="cri-o://1eb6e312303af46c6148f8d75082049119341a7297b369a98fcc70feeeb1b8ac" gracePeriod=45
Oct 31 01:04:42.042640 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:42.042602    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:42.043047 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:42.042721    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:04:42.991568 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:42.991526    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"]
Oct 31 01:04:47.750844 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:47.750809    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:48.554546 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:04:48.554501    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:04:51.770081 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:51.770001    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:04:51.775799 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:51.775777    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:04:52.043448 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:52.043367    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:57.750910 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:57.750874    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:04:58.011732 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:04:58.011649    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main" containerID="cri-o://d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405" gracePeriod=45
Oct 31 01:05:02.042711 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:02.042671    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:04.449017 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.448987    4888 generic.go:356] "Generic (PLEG): container finished" podID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerID="1eb6e312303af46c6148f8d75082049119341a7297b369a98fcc70feeeb1b8ac" exitCode=0
Oct 31 01:05:04.449402 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.449062    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" event={"ID":"7ed6435e-dd31-4b1d-a92c-d7838f6b280b","Type":"ContainerDied","Data":"1eb6e312303af46c6148f8d75082049119341a7297b369a98fcc70feeeb1b8ac"}
Oct 31 01:05:04.496599 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.496574    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:05:04.540964 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.540907    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-kube-api-access-5ckd2\" (UniqueName: \"kubernetes.io/projected/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-kube-api-access-5ckd2\") pod \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") "
Oct 31 01:05:04.540964 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.540937    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-tls-certs\") pod \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\" (UID: \"7ed6435e-dd31-4b1d-a92c-d7838f6b280b\") "
Oct 31 01:05:04.543110 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.543081    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-tls-certs" pod "7ed6435e-dd31-4b1d-a92c-d7838f6b280b" (UID: "7ed6435e-dd31-4b1d-a92c-d7838f6b280b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:04.543210 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.543163    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-kube-api-access-5ckd2" pod "7ed6435e-dd31-4b1d-a92c-d7838f6b280b" (UID: "7ed6435e-dd31-4b1d-a92c-d7838f6b280b"). InnerVolumeSpecName "kube-api-access-5ckd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:04.642112 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.642084    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ckd2\" (UniqueName: \"kubernetes.io/projected/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-kube-api-access-5ckd2\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:04.642112 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:04.642112    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ed6435e-dd31-4b1d-a92c-d7838f6b280b-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:05.455151 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:05.455113    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m" event={"ID":"7ed6435e-dd31-4b1d-a92c-d7838f6b280b","Type":"ContainerDied","Data":"5ed400cea708be5209aa7fbc6932f61a682c008bb6981fb330b3ffc8494660f7"}
Oct 31 01:05:05.455151 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:05.455135    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"
Oct 31 01:05:05.455657 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:05.455186    4888 scope.go:122] "RemoveContainer" containerID="1eb6e312303af46c6148f8d75082049119341a7297b369a98fcc70feeeb1b8ac"
Oct 31 01:05:05.614234 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:05.614205    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"]
Oct 31 01:05:05.660980 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:05.660948    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheb6w8m"]
Oct 31 01:05:07.554367 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:07.554320    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" path="/var/lib/kubelet/pods/7ed6435e-dd31-4b1d-a92c-d7838f6b280b/volumes"
Oct 31 01:05:07.752504 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:07.752445    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:07.752726 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:07.752565    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:05:17.751424 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:17.751387    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:18.551766 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:05:18.551725    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:05:27.750817 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:27.750776    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:05:28.245046 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.245021    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:05:28.322075 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.322003    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-tls-certs\") pod \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") "
Oct 31 01:05:28.322075 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.322037    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-kube-api-access-kwvln\" (UniqueName: \"kubernetes.io/projected/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-kube-api-access-kwvln\") pod \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\" (UID: \"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a\") "
Oct 31 01:05:28.324412 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.324379    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-tls-certs" pod "5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" (UID: "5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:05:28.324524 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.324416    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-kube-api-access-kwvln" pod "5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" (UID: "5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a"). InnerVolumeSpecName "kube-api-access-kwvln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:28.423234 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.423208    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:28.423234 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.423230    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwvln\" (UniqueName: \"kubernetes.io/projected/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a-kube-api-access-kwvln\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:05:28.556158 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.556124    4888 generic.go:356] "Generic (PLEG): container finished" podID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerID="d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405" exitCode=0
Oct 31 01:05:28.556311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.556170    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" event={"ID":"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a","Type":"ContainerDied","Data":"d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405"}
Oct 31 01:05:28.556311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.556175    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"
Oct 31 01:05:28.556311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.556207    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn" event={"ID":"5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a","Type":"ContainerDied","Data":"e0d5429f34b0bfed687574c7ad4fdaa7e8741e9df0d1f43f149e0c26edeffe85"}
Oct 31 01:05:28.556311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.556236    4888 scope.go:122] "RemoveContainer" containerID="d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405"
Oct 31 01:05:28.565246 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.565229    4888 scope.go:122] "RemoveContainer" containerID="d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405"
Oct 31 01:05:28.565542 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:05:28.565522    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405\": container with ID starting with d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405 not found: ID does not exist" containerID="d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405"
Oct 31 01:05:28.565594 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.565552    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405"} err="failed to get container status \"d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405\": rpc error: code = NotFound desc = could not find container \"d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405\": container with ID starting with d40020bebc7456d5943f5ca92d47a43e92c63e9c62bdd5b4b49532e451ddf405 not found: ID does not exist"
Oct 31 01:05:28.714639 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.714608    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"]
Oct 31 01:05:28.722044 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:28.722019    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schevj5rn"]
Oct 31 01:05:29.553818 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:29.553787    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" path="/var/lib/kubelet/pods/5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a/volumes"
Oct 31 01:05:38.436401 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.436368    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"]
Oct 31 01:05:38.436874 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.436859    4888 cpu_manager.go:414] "containerMap: removing container" podUID="a198573c-1780-422c-a752-279369b14bc7" containerName="main"
Oct 31 01:05:38.436931 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.436878    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a198573c-1780-422c-a752-279369b14bc7" containerName="main"
Oct 31 01:05:38.436931 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.436904    4888 cpu_manager.go:414] "containerMap: removing container" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main"
Oct 31 01:05:38.436931 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.436912    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main"
Oct 31 01:05:38.437035 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.436936    4888 cpu_manager.go:414] "containerMap: removing container" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main"
Oct 31 01:05:38.437035 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.436943    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main"
Oct 31 01:05:38.437035 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.437015    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="a198573c-1780-422c-a752-279369b14bc7" containerName="main"
Oct 31 01:05:38.437035 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.437029    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fb5c4b9-0ce3-44a3-9261-c6bc66d9aa6a" containerName="main"
Oct 31 01:05:38.437165 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.437040    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed6435e-dd31-4b1d-a92c-d7838f6b280b" containerName="main"
Oct 31 01:05:38.440599 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.440580    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.491007 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.490971    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"]
Oct 31 01:05:38.535995 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.535952    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Oct 31 01:05:38.535995 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.535974    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Oct 31 01:05:38.535995 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.535973    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-57nhq\""
Oct 31 01:05:38.536271 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.536146    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Oct 31 01:05:38.607621 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.607585    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.607788 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.607625    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v68xs\" (UniqueName: \"kubernetes.io/projected/87a09962-81ec-4eb1-b149-5f9f053bc420-kube-api-access-v68xs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.607788 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.607707    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.607903 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.607784    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a09962-81ec-4eb1-b149-5f9f053bc420-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.607903 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.607807    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.607903 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.607830    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.608015 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.607906    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709263 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709233    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709447 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709270    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709447 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709328    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709447 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709362    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v68xs\" (UniqueName: \"kubernetes.io/projected/87a09962-81ec-4eb1-b149-5f9f053bc420-kube-api-access-v68xs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709447 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709397    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709447 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709429    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a09962-81ec-4eb1-b149-5f9f053bc420-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709447 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709443    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709734 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709703    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709794 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709718    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709794 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709769    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.709936 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.709915    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.711867 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.711844    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.712018 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.711999    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a09962-81ec-4eb1-b149-5f9f053bc420-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.769314 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.769279    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v68xs\" (UniqueName: \"kubernetes.io/projected/87a09962-81ec-4eb1-b149-5f9f053bc420-kube-api-access-v68xs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:38.874961 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.874928    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"]
Oct 31 01:05:38.878617 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.878594    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:38.929441 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.929408    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-ldkrq\""
Oct 31 01:05:38.934442 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:38.934418    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"]
Oct 31 01:05:39.012005 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.011924    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mj9j\" (UniqueName: \"kubernetes.io/projected/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-kube-api-access-2mj9j\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:39.012005 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.011970    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:39.058730 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.058692    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:39.112963 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.112923    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mj9j\" (UniqueName: \"kubernetes.io/projected/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-kube-api-access-2mj9j\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:39.113137 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.113005    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:39.115892 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.115838    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:39.173822 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.173791    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mj9j\" (UniqueName: \"kubernetes.io/projected/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-kube-api-access-2mj9j\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:39.191266 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.191243    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:39.538507 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.538471    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"]
Oct 31 01:05:39.592956 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.592928    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"]
Oct 31 01:05:39.601232 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:05:39.601204    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4d1d6f_6a5a_41a0_a54f_c775f3a19da0.slice/crio-d07781440199d541bf942130a1660ffb7ed69cc5f74a555a767f8fcbbdfcbbd9 WatchSource:0}: Error finding container d07781440199d541bf942130a1660ffb7ed69cc5f74a555a767f8fcbbdfcbbd9: Status 404 returned error can't find the container with id d07781440199d541bf942130a1660ffb7ed69cc5f74a555a767f8fcbbdfcbbd9
Oct 31 01:05:39.608935 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.608908    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" event={"ID":"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0","Type":"ContainerStarted","Data":"d07781440199d541bf942130a1660ffb7ed69cc5f74a555a767f8fcbbdfcbbd9"}
Oct 31 01:05:39.629942 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:39.629922    4888 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:05:40.615091 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:40.615057    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" event={"ID":"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0","Type":"ContainerStarted","Data":"6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203"}
Oct 31 01:05:40.615562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:40.615158    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:05:40.616611 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:40.616579    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerStarted","Data":"82b7158cf5f8847b91b88378567361482821006f07070955a2be07dc923245f1"}
Oct 31 01:05:40.728590 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:40.728539    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" podStartSLOduration=2.7285225410000002 podStartE2EDuration="2.728522541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:40.728301198 +0000 UTC m=+2149.283188636" watchObservedRunningTime="2026-10-31 01:05:40.728522541 +0000 UTC m=+2149.283409977"
Oct 31 01:05:47.657338 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:47.657310    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerStarted","Data":"0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86"}
Oct 31 01:05:48.552001 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:05:48.551967    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:05:49.667831 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:49.667743    4888 generic.go:356] "Generic (PLEG): container finished" podID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerID="0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86" exitCode=0
Oct 31 01:05:49.667831 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:49.667818    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerDied","Data":"0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86"}
Oct 31 01:05:50.675060 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:50.675025    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerStarted","Data":"c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c"}
Oct 31 01:05:51.682188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:51.682152    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerStarted","Data":"cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c"}
Oct 31 01:05:51.895297 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:51.895252    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podStartSLOduration=4.296636369 podStartE2EDuration="13.895238142s" totalImagesPullingTime="8.51226517s" totalInitContainerRuntime="1.086336603s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:51.846784603 +0000 UTC m=+2160.401672064" watchObservedRunningTime="2026-10-31 01:05:51.895238142 +0000 UTC m=+2160.450125579"
Oct 31 01:05:52.059748 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:52.059713    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:59.059256 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:59.059220    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:05:59.060612 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:05:59.060581    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:06:09.059256 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:06:09.059210    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:06:11.624638 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:06:11.624607    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:06:18.558817 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:06:18.558780    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:06:19.060087 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:06:19.060048    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:06:29.059398 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:06:29.059297    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:06:39.060121 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:06:39.060077    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:06:48.552813 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:06:48.552775    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:06:49.059512 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:06:49.059471    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:06:59.059811 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:06:59.059773    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:07:09.059342 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:09.059298    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:07:18.552675 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:07:18.552635    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:07:19.059720 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:19.059673    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:07:29.059880 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:29.059837    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Oct 31 01:07:39.069366 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:39.069314    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:07:39.076894 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:39.076873    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:07:48.042486 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:48.042455    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"]
Oct 31 01:07:48.077646 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:48.077616    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"]
Oct 31 01:07:48.077900 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:48.077869    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="modelcar" containerID="cri-o://cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c" gracePeriod=60
Oct 31 01:07:48.274676 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:48.274648    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl_87a09962-81ec-4eb1-b149-5f9f053bc420/modelcar/0.log"
Oct 31 01:07:48.275299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:48.275274    4888 generic.go:356] "Generic (PLEG): container finished" podID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerID="cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c" exitCode=143
Oct 31 01:07:48.275448 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:48.275331    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerDied","Data":"cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c"}
Oct 31 01:07:48.552620 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:07:48.552583    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:07:51.624779 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:07:51.624681    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:01.624325 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:01.624281    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:03.062134 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.062077    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main" containerID="cri-o://6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203" gracePeriod=45
Oct 31 01:08:03.101084 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.101051    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main" containerID="cri-o://c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c" gracePeriod=45
Oct 31 01:08:03.874621 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.874599    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl_87a09962-81ec-4eb1-b149-5f9f053bc420/modelcar/0.log"
Oct 31 01:08:03.877517 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.877497    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:08:03.969594 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969567    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-kserve-provision-location\") pod \"87a09962-81ec-4eb1-b149-5f9f053bc420\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") "
Oct 31 01:08:03.969737 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969629    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87a09962-81ec-4eb1-b149-5f9f053bc420-tls-certs\" (UniqueName: \"kubernetes.io/secret/87a09962-81ec-4eb1-b149-5f9f053bc420-tls-certs\") pod \"87a09962-81ec-4eb1-b149-5f9f053bc420\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") "
Oct 31 01:08:03.969737 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969707    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-tmp-dir\") pod \"87a09962-81ec-4eb1-b149-5f9f053bc420\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") "
Oct 31 01:08:03.969808 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969753    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-home\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-home\") pod \"87a09962-81ec-4eb1-b149-5f9f053bc420\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") "
Oct 31 01:08:03.969808 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969778    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-model-cache\") pod \"87a09962-81ec-4eb1-b149-5f9f053bc420\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") "
Oct 31 01:08:03.969912 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969804    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-dshm\") pod \"87a09962-81ec-4eb1-b149-5f9f053bc420\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") "
Oct 31 01:08:03.969912 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969836    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87a09962-81ec-4eb1-b149-5f9f053bc420-kube-api-access-v68xs\" (UniqueName: \"kubernetes.io/projected/87a09962-81ec-4eb1-b149-5f9f053bc420-kube-api-access-v68xs\") pod \"87a09962-81ec-4eb1-b149-5f9f053bc420\" (UID: \"87a09962-81ec-4eb1-b149-5f9f053bc420\") "
Oct 31 01:08:03.970012 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.969927    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-kserve-provision-location" pod "87a09962-81ec-4eb1-b149-5f9f053bc420" (UID: "87a09962-81ec-4eb1-b149-5f9f053bc420"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:03.970097 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.970057    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-model-cache" pod "87a09962-81ec-4eb1-b149-5f9f053bc420" (UID: "87a09962-81ec-4eb1-b149-5f9f053bc420"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:03.970097 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.970081    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:03.970571 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.970536    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-home" pod "87a09962-81ec-4eb1-b149-5f9f053bc420" (UID: "87a09962-81ec-4eb1-b149-5f9f053bc420"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:03.972091 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.972058    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87a09962-81ec-4eb1-b149-5f9f053bc420-kube-api-access-v68xs" pod "87a09962-81ec-4eb1-b149-5f9f053bc420" (UID: "87a09962-81ec-4eb1-b149-5f9f053bc420"). InnerVolumeSpecName "kube-api-access-v68xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:03.972174 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.972096    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87a09962-81ec-4eb1-b149-5f9f053bc420-tls-certs" pod "87a09962-81ec-4eb1-b149-5f9f053bc420" (UID: "87a09962-81ec-4eb1-b149-5f9f053bc420"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:03.972174 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.972106    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-dshm" pod "87a09962-81ec-4eb1-b149-5f9f053bc420" (UID: "87a09962-81ec-4eb1-b149-5f9f053bc420"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:03.980701 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:03.980677    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-tmp-dir" pod "87a09962-81ec-4eb1-b149-5f9f053bc420" (UID: "87a09962-81ec-4eb1-b149-5f9f053bc420"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:04.071043 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.071018    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:04.071043 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.071039    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:04.071402 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.071047    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:04.071402 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.071057    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87a09962-81ec-4eb1-b149-5f9f053bc420-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:04.071402 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.071066    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v68xs\" (UniqueName: \"kubernetes.io/projected/87a09962-81ec-4eb1-b149-5f9f053bc420-kube-api-access-v68xs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:04.071402 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.071073    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87a09962-81ec-4eb1-b149-5f9f053bc420-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:04.354160 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.354133    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl_87a09962-81ec-4eb1-b149-5f9f053bc420/modelcar/0.log"
Oct 31 01:08:04.354816 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.354796    4888 generic.go:356] "Generic (PLEG): container finished" podID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerID="c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c" exitCode=0
Oct 31 01:08:04.354925 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.354865    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"
Oct 31 01:08:04.354925 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.354882    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerDied","Data":"c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c"}
Oct 31 01:08:04.355014 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.354941    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl" event={"ID":"87a09962-81ec-4eb1-b149-5f9f053bc420","Type":"ContainerDied","Data":"82b7158cf5f8847b91b88378567361482821006f07070955a2be07dc923245f1"}
Oct 31 01:08:04.355014 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.354969    4888 scope.go:122] "RemoveContainer" containerID="cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c"
Oct 31 01:08:04.364664 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.364648    4888 scope.go:122] "RemoveContainer" containerID="c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c"
Oct 31 01:08:04.372222 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.372203    4888 scope.go:122] "RemoveContainer" containerID="0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86"
Oct 31 01:08:04.379784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.379769    4888 scope.go:122] "RemoveContainer" containerID="cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c"
Oct 31 01:08:04.380046 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:08:04.380025    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c\": container with ID starting with cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c not found: ID does not exist" containerID="cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c"
Oct 31 01:08:04.380095 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.380056    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c"} err="failed to get container status \"cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c\": rpc error: code = NotFound desc = could not find container \"cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c\": container with ID starting with cfb8ce1f9c7f0cc314c5f23632c0c12fb0ff9f733de28b3395794a229de3ca3c not found: ID does not exist"
Oct 31 01:08:04.380095 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.380073    4888 scope.go:122] "RemoveContainer" containerID="c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c"
Oct 31 01:08:04.380292 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:08:04.380275    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c\": container with ID starting with c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c not found: ID does not exist" containerID="c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c"
Oct 31 01:08:04.380333 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.380302    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c"} err="failed to get container status \"c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c\": rpc error: code = NotFound desc = could not find container \"c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c\": container with ID starting with c5b93f419cd908e57401f630e7b85abcdb9430949f23665c0d4a64463129937c not found: ID does not exist"
Oct 31 01:08:04.380333 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.380319    4888 scope.go:122] "RemoveContainer" containerID="0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86"
Oct 31 01:08:04.380542 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:08:04.380524    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86\": container with ID starting with 0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86 not found: ID does not exist" containerID="0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86"
Oct 31 01:08:04.380580 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.380547    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86"} err="failed to get container status \"0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86\": rpc error: code = NotFound desc = could not find container \"0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86\": container with ID starting with 0948aa889a0194bb94ad192d4d2ac40128712571fe014e3a9bce16ef121fbf86 not found: ID does not exist"
Oct 31 01:08:04.516026 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.515998    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"]
Oct 31 01:08:04.562950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:04.562922    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-7687c48f9bq98wl"]
Oct 31 01:08:05.553225 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:05.553192    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" path="/var/lib/kubelet/pods/87a09962-81ec-4eb1-b149-5f9f053bc420/volumes"
Oct 31 01:08:11.623630 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:11.623588    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:11.624008 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:11.623693    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:08:18.551755 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:08:18.551722    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:08:21.626496 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:21.626444    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:31.624237 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:31.624202    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:33.316444 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.316418    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:08:33.487457 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.487427    4888 generic.go:356] "Generic (PLEG): container finished" podID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerID="6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203" exitCode=0
Oct 31 01:08:33.487630 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.487486    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"
Oct 31 01:08:33.487630 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.487518    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" event={"ID":"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0","Type":"ContainerDied","Data":"6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203"}
Oct 31 01:08:33.487630 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.487604    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz" event={"ID":"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0","Type":"ContainerDied","Data":"d07781440199d541bf942130a1660ffb7ed69cc5f74a555a767f8fcbbdfcbbd9"}
Oct 31 01:08:33.487735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.487630    4888 scope.go:122] "RemoveContainer" containerID="6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203"
Oct 31 01:08:33.495746 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.495728    4888 scope.go:122] "RemoveContainer" containerID="6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203"
Oct 31 01:08:33.496027 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:08:33.496008    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203\": container with ID starting with 6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203 not found: ID does not exist" containerID="6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203"
Oct 31 01:08:33.496089 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.496036    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203"} err="failed to get container status \"6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203\": rpc error: code = NotFound desc = could not find container \"6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203\": container with ID starting with 6c480d69380fc8ff523680a755aa318314a226de0ae3b214c1653d0afbddf203 not found: ID does not exist"
Oct 31 01:08:33.506445 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.506425    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-kube-api-access-2mj9j\" (UniqueName: \"kubernetes.io/projected/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-kube-api-access-2mj9j\") pod \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") "
Oct 31 01:08:33.506547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.506475    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-tls-certs\") pod \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\" (UID: \"ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0\") "
Oct 31 01:08:33.508490 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.508467    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-kube-api-access-2mj9j" pod "ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" (UID: "ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0"). InnerVolumeSpecName "kube-api-access-2mj9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:33.508590 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.508491    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-tls-certs" pod "ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" (UID: "ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:33.607196 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.607172    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:33.607296 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.607198    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mj9j\" (UniqueName: \"kubernetes.io/projected/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0-kube-api-access-2mj9j\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:08:33.898748 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.898670    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"]
Oct 31 01:08:33.991966 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:33.991933    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-sche2cbcz"]
Oct 31 01:08:35.553188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:35.553148    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" path="/var/lib/kubelet/pods/ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0/volumes"
Oct 31 01:08:48.551705 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:08:48.551663    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:08:52.492087 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492054    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"]
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492390    4888 cpu_manager.go:414] "containerMap: removing container" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492404    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492422    4888 cpu_manager.go:414] "containerMap: removing container" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492426    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492435    4888 cpu_manager.go:414] "containerMap: removing container" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="modelcar-init"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492439    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="modelcar-init"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492456    4888 cpu_manager.go:414] "containerMap: removing container" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="modelcar"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492461    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="modelcar"
Oct 31 01:08:52.492516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492516    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="main"
Oct 31 01:08:52.492819 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492526    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="87a09962-81ec-4eb1-b149-5f9f053bc420" containerName="modelcar"
Oct 31 01:08:52.492819 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.492533    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4d1d6f-6a5a-41a0-a54f-c775f3a19da0" containerName="main"
Oct 31 01:08:52.496911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.496894    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.547572 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.547520    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Oct 31 01:08:52.547725 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.547638    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-dzgtx\""
Oct 31 01:08:52.547799 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.547785    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Oct 31 01:08:52.557161 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557136    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-dshm\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.557281 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557165    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln8g2\" (UniqueName: \"kubernetes.io/projected/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kube-api-access-ln8g2\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.557281 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557198    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tls-certs\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.557281 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557268    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33c8ec1-0881-4bae-964e-8a33d648b3d4-cabundle-cert\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.557407 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557308    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-model-cache\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.557407 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557392    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.557475 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557420    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tmp-dir\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.557475 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.557452    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-home\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.594478 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.594448    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Oct 31 01:08:52.594478 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.594448    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Oct 31 01:08:52.594674 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.594554    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:08:52.596485 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.596465    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"]
Oct 31 01:08:52.658204 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658176    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-dshm\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658204 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658205    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln8g2\" (UniqueName: \"kubernetes.io/projected/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kube-api-access-ln8g2\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658461 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658232    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tls-certs\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658461 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658309    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33c8ec1-0881-4bae-964e-8a33d648b3d4-cabundle-cert\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658461 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658378    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-model-cache\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658461 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658407    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658461 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658432    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tmp-dir\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658461 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658460    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-home\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.658996 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.658972    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-model-cache\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.659182 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.659157    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-home\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.659649 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.659412    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tmp-dir\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.659734 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.659662    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.659940 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.659925    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33c8ec1-0881-4bae-964e-8a33d648b3d4-cabundle-cert\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.660718 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.660699    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-dshm\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.660861 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.660841    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tls-certs\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.718135 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.718098    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln8g2\" (UniqueName: \"kubernetes.io/projected/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kube-api-access-ln8g2\") pod \"custom-route-timeout-test-kserve-bfbf46f68-qrdz9\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:52.813379 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:52.813261    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:53.158383 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.158293    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"]
Oct 31 01:08:53.162642 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.162624    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.259999 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.259947    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-wdn72\""
Oct 31 01:08:53.259999 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.259966    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-n8fmm\""
Oct 31 01:08:53.259999 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.259973    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Oct 31 01:08:53.259999 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.259976    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Oct 31 01:08:53.260508 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.260489    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Oct 31 01:08:53.262598 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.262578    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"]
Oct 31 01:08:53.262956 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.262935    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchkx\" (UniqueName: \"kubernetes.io/projected/793d2c64-3fb3-4c25-8457-fa53c1fecc41-kube-api-access-gchkx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.263024 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.263006    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d2c64-3fb3-4c25-8457-fa53c1fecc41-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.263915 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:08:53.263884    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb33c8ec1_0881_4bae_964e_8a33d648b3d4.slice/crio-9e0bae6bd9d090f0d231b7144b66576e148bdd1083345093b2aaacc19fd40794 WatchSource:0}: Error finding container 9e0bae6bd9d090f0d231b7144b66576e148bdd1083345093b2aaacc19fd40794: Status 404 returned error can't find the container with id 9e0bae6bd9d090f0d231b7144b66576e148bdd1083345093b2aaacc19fd40794
Oct 31 01:08:53.308786 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.308766    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"]
Oct 31 01:08:53.364026 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.363977    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchkx\" (UniqueName: \"kubernetes.io/projected/793d2c64-3fb3-4c25-8457-fa53c1fecc41-kube-api-access-gchkx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.364188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.364092    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d2c64-3fb3-4c25-8457-fa53c1fecc41-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.366594 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.366574    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d2c64-3fb3-4c25-8457-fa53c1fecc41-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.425430 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.424853    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchkx\" (UniqueName: \"kubernetes.io/projected/793d2c64-3fb3-4c25-8457-fa53c1fecc41-kube-api-access-gchkx\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.475263 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.475230    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:53.584162 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.584121    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" event={"ID":"b33c8ec1-0881-4bae-964e-8a33d648b3d4","Type":"ContainerStarted","Data":"95094efcfbfadafc716b8616c16360672ab486d079e7f5ad15ef9709d92b3bd9"}
Oct 31 01:08:53.584667 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.584179    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" event={"ID":"b33c8ec1-0881-4bae-964e-8a33d648b3d4","Type":"ContainerStarted","Data":"9e0bae6bd9d090f0d231b7144b66576e148bdd1083345093b2aaacc19fd40794"}
Oct 31 01:08:53.849821 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:53.849784    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"]
Oct 31 01:08:53.857645 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:08:53.857614    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793d2c64_3fb3_4c25_8457_fa53c1fecc41.slice/crio-d87a3d6ba6e9b20f3404d2356a75a96dbbdaa53bb184ac9a485cc85758b6ee3d WatchSource:0}: Error finding container d87a3d6ba6e9b20f3404d2356a75a96dbbdaa53bb184ac9a485cc85758b6ee3d: Status 404 returned error can't find the container with id d87a3d6ba6e9b20f3404d2356a75a96dbbdaa53bb184ac9a485cc85758b6ee3d
Oct 31 01:08:54.590131 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:54.590084    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" event={"ID":"793d2c64-3fb3-4c25-8457-fa53c1fecc41","Type":"ContainerStarted","Data":"023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb"}
Oct 31 01:08:54.590629 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:54.590150    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" event={"ID":"793d2c64-3fb3-4c25-8457-fa53c1fecc41","Type":"ContainerStarted","Data":"d87a3d6ba6e9b20f3404d2356a75a96dbbdaa53bb184ac9a485cc85758b6ee3d"}
Oct 31 01:08:54.590629 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:54.590373    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:08:54.703218 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:54.703170    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" podStartSLOduration=1.7031571840000002 podStartE2EDuration="1.703157184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:54.701452726 +0000 UTC m=+2343.256340163" watchObservedRunningTime="2026-10-31 01:08:54.703157184 +0000 UTC m=+2343.258044621"
Oct 31 01:08:55.595631 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:55.595598    4888 generic.go:356] "Generic (PLEG): container finished" podID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerID="95094efcfbfadafc716b8616c16360672ab486d079e7f5ad15ef9709d92b3bd9" exitCode=0
Oct 31 01:08:55.596081 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:55.595674    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" event={"ID":"b33c8ec1-0881-4bae-964e-8a33d648b3d4","Type":"ContainerDied","Data":"95094efcfbfadafc716b8616c16360672ab486d079e7f5ad15ef9709d92b3bd9"}
Oct 31 01:08:56.602575 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:56.602542    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" event={"ID":"b33c8ec1-0881-4bae-964e-8a33d648b3d4","Type":"ContainerStarted","Data":"96bf21c268ac964f35d7ece5292e411439fe51c8b8e99198d8a07d79401a5c4c"}
Oct 31 01:08:56.814060 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:56.814027    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:08:56.817129 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:08:56.817073    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podStartSLOduration=2.6989739569999998 podStartE2EDuration="4.817057799s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.118083842s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:08:56.767626326 +0000 UTC m=+2345.322513799" watchObservedRunningTime="2026-10-31 01:08:56.817057799 +0000 UTC m=+2345.371945239"
Oct 31 01:09:02.818374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:02.816982    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:09:02.818374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:02.817882    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:09:12.813996 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:12.813955    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:09:18.551368 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:09:18.551322    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:09:22.814225 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:22.814122    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:09:25.597302 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:25.597275    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:09:32.814584 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:32.814539    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:09:42.814590 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:42.814535    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:09:48.552743 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:09:48.552709    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:09:51.802304 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:51.802273    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:09:51.810648 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:51.810628    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:09:52.814324 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:09:52.814286    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:10:02.813827 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:02.813778    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:10:12.814385 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:12.814322    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:10:16.483386 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:16.483332    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"]
Oct 31 01:10:18.552114 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:10:18.552077    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:10:22.814038 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:22.813993    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:10:25.597837 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:25.597794    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:10:31.502998 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:31.502947    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main" containerID="cri-o://023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb" gracePeriod=45
Oct 31 01:10:32.813834 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:32.813779    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:10:35.597462 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:35.597427    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:10:42.823682 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:42.823651    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:10:42.830987 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:42.830962    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:10:45.597878 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:45.597845    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:10:45.598256 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:45.597955    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:10:48.551614 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:10:48.551572    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:10:51.644164 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:51.644091    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"]
Oct 31 01:10:55.598092 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:10:55.598053    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:11:01.746335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:01.746308    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:11:01.850707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:01.850626    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793d2c64-3fb3-4c25-8457-fa53c1fecc41-kube-api-access-gchkx\" (UniqueName: \"kubernetes.io/projected/793d2c64-3fb3-4c25-8457-fa53c1fecc41-kube-api-access-gchkx\") pod \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") "
Oct 31 01:11:01.850707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:01.850668    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/793d2c64-3fb3-4c25-8457-fa53c1fecc41-tls-certs\" (UniqueName: \"kubernetes.io/secret/793d2c64-3fb3-4c25-8457-fa53c1fecc41-tls-certs\") pod \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\" (UID: \"793d2c64-3fb3-4c25-8457-fa53c1fecc41\") "
Oct 31 01:11:01.853044 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:01.853011    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793d2c64-3fb3-4c25-8457-fa53c1fecc41-kube-api-access-gchkx" pod "793d2c64-3fb3-4c25-8457-fa53c1fecc41" (UID: "793d2c64-3fb3-4c25-8457-fa53c1fecc41"). InnerVolumeSpecName "kube-api-access-gchkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:01.853179 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:01.853041    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/793d2c64-3fb3-4c25-8457-fa53c1fecc41-tls-certs" pod "793d2c64-3fb3-4c25-8457-fa53c1fecc41" (UID: "793d2c64-3fb3-4c25-8457-fa53c1fecc41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:01.951138 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:01.951110    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchkx\" (UniqueName: \"kubernetes.io/projected/793d2c64-3fb3-4c25-8457-fa53c1fecc41-kube-api-access-gchkx\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:01.951138 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:01.951134    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/793d2c64-3fb3-4c25-8457-fa53c1fecc41-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:02.261009 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.260978    4888 generic.go:356] "Generic (PLEG): container finished" podID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerID="023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb" exitCode=0
Oct 31 01:11:02.261171 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.261030    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"
Oct 31 01:11:02.261171 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.261048    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" event={"ID":"793d2c64-3fb3-4c25-8457-fa53c1fecc41","Type":"ContainerDied","Data":"023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb"}
Oct 31 01:11:02.261171 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.261106    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46" event={"ID":"793d2c64-3fb3-4c25-8457-fa53c1fecc41","Type":"ContainerDied","Data":"d87a3d6ba6e9b20f3404d2356a75a96dbbdaa53bb184ac9a485cc85758b6ee3d"}
Oct 31 01:11:02.261171 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.261135    4888 scope.go:122] "RemoveContainer" containerID="023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb"
Oct 31 01:11:02.270584 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.270569    4888 scope.go:122] "RemoveContainer" containerID="023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb"
Oct 31 01:11:02.270845 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:11:02.270828    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb\": container with ID starting with 023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb not found: ID does not exist" containerID="023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb"
Oct 31 01:11:02.270886 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.270854    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb"} err="failed to get container status \"023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb\": rpc error: code = NotFound desc = could not find container \"023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb\": container with ID starting with 023e3168c869e204781ba6497ac905076a520a5f528fb3e35c0a0fde2bc10ccb not found: ID does not exist"
Oct 31 01:11:02.419979 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.419950    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"]
Oct 31 01:11:02.428805 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:02.428778    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-sche6zl46"]
Oct 31 01:11:03.553274 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:03.553241    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" path="/var/lib/kubelet/pods/793d2c64-3fb3-4c25-8457-fa53c1fecc41/volumes"
Oct 31 01:11:06.666539 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:06.666480    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" containerID="cri-o://96bf21c268ac964f35d7ece5292e411439fe51c8b8e99198d8a07d79401a5c4c" gracePeriod=45
Oct 31 01:11:06.825217 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:06.825176    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 31 01:11:07.285270 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.285240    4888 generic.go:356] "Generic (PLEG): container finished" podID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerID="96bf21c268ac964f35d7ece5292e411439fe51c8b8e99198d8a07d79401a5c4c" exitCode=0
Oct 31 01:11:07.285398 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.285284    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" event={"ID":"b33c8ec1-0881-4bae-964e-8a33d648b3d4","Type":"ContainerDied","Data":"96bf21c268ac964f35d7ece5292e411439fe51c8b8e99198d8a07d79401a5c4c"}
Oct 31 01:11:07.413722 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.413695    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:11:07.497866 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.497837    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kserve-provision-location\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498048 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.497876    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tls-certs\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498048 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.497896    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-dshm\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498048 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.497910    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-home\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-home\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498048 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.497957    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tmp-dir\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498048 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.498017    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kube-api-access-ln8g2\" (UniqueName: \"kubernetes.io/projected/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kube-api-access-ln8g2\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498048 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.498042    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33c8ec1-0881-4bae-964e-8a33d648b3d4-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33c8ec1-0881-4bae-964e-8a33d648b3d4-cabundle-cert\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498408 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.498087    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-model-cache\") pod \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\" (UID: \"b33c8ec1-0881-4bae-964e-8a33d648b3d4\") "
Oct 31 01:11:07.498530 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.498506    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-model-cache" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:07.498852 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.498652    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33c8ec1-0881-4bae-964e-8a33d648b3d4-cabundle-cert" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:07.498852 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.498768    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-home" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:07.500422 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.500397    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tls-certs" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:07.500527 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.500486    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-dshm" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:07.500632 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.500608    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kube-api-access-ln8g2" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "kube-api-access-ln8g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:07.515995 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.515967    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tmp-dir" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:07.526441 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.526418    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kserve-provision-location" pod "b33c8ec1-0881-4bae-964e-8a33d648b3d4" (UID: "b33c8ec1-0881-4bae-964e-8a33d648b3d4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:07.599367 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599324    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.599367 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599359    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.599526 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599371    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.599526 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599385    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.599526 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599398    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln8g2\" (UniqueName: \"kubernetes.io/projected/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kube-api-access-ln8g2\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.599526 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599409    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b33c8ec1-0881-4bae-964e-8a33d648b3d4-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.599526 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599418    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:07.599526 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:07.599426    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b33c8ec1-0881-4bae-964e-8a33d648b3d4-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:11:08.291999 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:08.291974    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"
Oct 31 01:11:08.292479 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:08.291976    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9" event={"ID":"b33c8ec1-0881-4bae-964e-8a33d648b3d4","Type":"ContainerDied","Data":"9e0bae6bd9d090f0d231b7144b66576e148bdd1083345093b2aaacc19fd40794"}
Oct 31 01:11:08.292479 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:08.292103    4888 scope.go:122] "RemoveContainer" containerID="96bf21c268ac964f35d7ece5292e411439fe51c8b8e99198d8a07d79401a5c4c"
Oct 31 01:11:08.300261 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:08.300241    4888 scope.go:122] "RemoveContainer" containerID="95094efcfbfadafc716b8616c16360672ab486d079e7f5ad15ef9709d92b3bd9"
Oct 31 01:11:08.500222 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:08.500189    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"]
Oct 31 01:11:08.547248 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:08.547172    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-bfbf46f68-qrdz9"]
Oct 31 01:11:09.553214 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:09.553187    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" path="/var/lib/kubelet/pods/b33c8ec1-0881-4bae-964e-8a33d648b3d4/volumes"
Oct 31 01:11:14.165426 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165391    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"]
Oct 31 01:11:14.165804 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165710    4888 cpu_manager.go:414] "containerMap: removing container" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="storage-initializer"
Oct 31 01:11:14.165804 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165721    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="storage-initializer"
Oct 31 01:11:14.165804 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165731    4888 cpu_manager.go:414] "containerMap: removing container" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main"
Oct 31 01:11:14.165804 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165736    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main"
Oct 31 01:11:14.165804 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165752    4888 cpu_manager.go:414] "containerMap: removing container" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main"
Oct 31 01:11:14.165804 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165756    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main"
Oct 31 01:11:14.166001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165814    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="793d2c64-3fb3-4c25-8457-fa53c1fecc41" containerName="main"
Oct 31 01:11:14.166001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.165825    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33c8ec1-0881-4bae-964e-8a33d648b3d4" containerName="main"
Oct 31 01:11:14.170227 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.170192    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.222594 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.222564    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"]
Oct 31 01:11:14.253888 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.253860    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-dshm\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.254018 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.253894    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-tmp-dir\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.254018 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.253912    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-home\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.254018 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.253927    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8m7\" (UniqueName: \"kubernetes.io/projected/0b945160-2367-4c3d-be09-24c8ba425be6-kube-api-access-2s8m7\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.254139 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.254064    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-kserve-provision-location\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.254139 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.254096    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b945160-2367-4c3d-be09-24c8ba425be6-tls-certs\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.254139 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.254132    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-model-cache\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.254245 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.254218    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b945160-2367-4c3d-be09-24c8ba425be6-cabundle-cert\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.267279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.267246    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Oct 31 01:11:14.267279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.267243    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Oct 31 01:11:14.267279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.267266    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:11:14.267483 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.267271    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Oct 31 01:11:14.267483 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.267300    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Oct 31 01:11:14.267483 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.267341    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-jf94k\""
Oct 31 01:11:14.355452 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355423    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-kserve-provision-location\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355617 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355458    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b945160-2367-4c3d-be09-24c8ba425be6-tls-certs\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355617 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355502    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-model-cache\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355617 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355527    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b945160-2367-4c3d-be09-24c8ba425be6-cabundle-cert\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355617 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355562    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-dshm\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355617 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355594    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-tmp-dir\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355617 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355616    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-home\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355640    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8m7\" (UniqueName: \"kubernetes.io/projected/0b945160-2367-4c3d-be09-24c8ba425be6-kube-api-access-2s8m7\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355917    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-model-cache\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355940    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-tmp-dir\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.355988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.355973    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-kserve-provision-location\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.356188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.356079    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-home\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.356299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.356280    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b945160-2367-4c3d-be09-24c8ba425be6-cabundle-cert\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.357911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.357891    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-dshm\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.358271 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.358251    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b945160-2367-4c3d-be09-24c8ba425be6-tls-certs\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.416024 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.415950    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8m7\" (UniqueName: \"kubernetes.io/projected/0b945160-2367-4c3d-be09-24c8ba425be6-kube-api-access-2s8m7\") pod \"stop-feature-test-kserve-5678d64c4-8jdk9\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.485144 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:14.485108    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:14.974585 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:11:14.974549    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b945160_2367_4c3d_be09_24c8ba425be6.slice/crio-95fd52692bc8531dd75391710f2628bdcd7b68bd0e613ee1cd5c5bdd651203bd WatchSource:0}: Error finding container 95fd52692bc8531dd75391710f2628bdcd7b68bd0e613ee1cd5c5bdd651203bd: Status 404 returned error can't find the container with id 95fd52692bc8531dd75391710f2628bdcd7b68bd0e613ee1cd5c5bdd651203bd
Oct 31 01:11:15.013878 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:15.013851    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"]
Oct 31 01:11:15.326107 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:15.326022    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" event={"ID":"0b945160-2367-4c3d-be09-24c8ba425be6","Type":"ContainerStarted","Data":"a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d"}
Oct 31 01:11:15.326107 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:15.326082    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" event={"ID":"0b945160-2367-4c3d-be09-24c8ba425be6","Type":"ContainerStarted","Data":"95fd52692bc8531dd75391710f2628bdcd7b68bd0e613ee1cd5c5bdd651203bd"}
Oct 31 01:11:17.337673 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:17.337636    4888 generic.go:356] "Generic (PLEG): container finished" podID="0b945160-2367-4c3d-be09-24c8ba425be6" containerID="a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d" exitCode=0
Oct 31 01:11:17.338108 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:17.337729    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" event={"ID":"0b945160-2367-4c3d-be09-24c8ba425be6","Type":"ContainerDied","Data":"a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d"}
Oct 31 01:11:18.345041 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.345004    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" event={"ID":"0b945160-2367-4c3d-be09-24c8ba425be6","Type":"ContainerStarted","Data":"32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8"}
Oct 31 01:11:18.444689 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.444657    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"]
Oct 31 01:11:18.448245 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.448221    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:18.468263 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.468235    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"]
Oct 31 01:11:18.486249 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.486224    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:18.499746 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.499721    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-zqxmg\""
Oct 31 01:11:18.558489 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.558438    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podStartSLOduration=2.484534107 podStartE2EDuration="4.55842334s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.073889233s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:18.510640915 +0000 UTC m=+2487.065528353" watchObservedRunningTime="2026-10-31 01:11:18.55842334 +0000 UTC m=+2487.113310777"
Oct 31 01:11:18.564162 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:11:18.564130    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:11:18.599224 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.599128    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddb732-84a7-4108-917a-7cdd29e6680b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:18.599224 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.599191    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2w2z\" (UniqueName: \"kubernetes.io/projected/4eddb732-84a7-4108-917a-7cdd29e6680b-kube-api-access-p2w2z\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:18.700523 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.700493    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddb732-84a7-4108-917a-7cdd29e6680b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:18.700523 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.700526    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2w2z\" (UniqueName: \"kubernetes.io/projected/4eddb732-84a7-4108-917a-7cdd29e6680b-kube-api-access-p2w2z\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:18.703070 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.703040    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddb732-84a7-4108-917a-7cdd29e6680b-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:18.761565 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:18.761531    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2w2z\" (UniqueName: \"kubernetes.io/projected/4eddb732-84a7-4108-917a-7cdd29e6680b-kube-api-access-p2w2z\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:19.061814 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:19.061780    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:19.449343 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:19.449308    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"]
Oct 31 01:11:19.456033 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:11:19.456003    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eddb732_84a7_4108_917a_7cdd29e6680b.slice/crio-57f61b6e4608f827a297d1b7163f2c71b901e739fd164cb99f6ea23cf3fa895e WatchSource:0}: Error finding container 57f61b6e4608f827a297d1b7163f2c71b901e739fd164cb99f6ea23cf3fa895e: Status 404 returned error can't find the container with id 57f61b6e4608f827a297d1b7163f2c71b901e739fd164cb99f6ea23cf3fa895e
Oct 31 01:11:20.357297 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:20.357260    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" event={"ID":"4eddb732-84a7-4108-917a-7cdd29e6680b","Type":"ContainerStarted","Data":"fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff"}
Oct 31 01:11:20.357607 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:20.357417    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:20.357607 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:20.357461    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" event={"ID":"4eddb732-84a7-4108-917a-7cdd29e6680b","Type":"ContainerStarted","Data":"57f61b6e4608f827a297d1b7163f2c71b901e739fd164cb99f6ea23cf3fa895e"}
Oct 31 01:11:20.470907 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:20.470842    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" podStartSLOduration=2.470823464 podStartE2EDuration="2.470823464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:20.469886228 +0000 UTC m=+2489.024773668" watchObservedRunningTime="2026-10-31 01:11:20.470823464 +0000 UTC m=+2489.025710884"
Oct 31 01:11:24.486232 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:24.486189    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:11:24.487939 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:24.487909    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:11:34.486516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:34.486473    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:11:44.485780 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:44.485738    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:11:48.552261 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:11:48.552215    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:11:51.364424 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:51.364384    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:11:54.485986 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:11:54.485936    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:12:04.486579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:12:04.486522    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:12:14.486197 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:12:14.486149    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:12:18.553034 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:12:18.552996    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:12:24.485728 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:12:24.485628    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:12:34.486436 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:12:34.486393    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:12:44.485543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:12:44.485493    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:12:48.552954 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:12:48.552916    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:12:54.485846 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:12:54.485799    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:13:04.495441 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:04.495411    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:13:04.502758 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:04.502736    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:13:11.904090 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:11.904061    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"]
Oct 31 01:13:12.070012 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:12.069982    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"]
Oct 31 01:13:18.552489 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:13:18.552453    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:13:21.364442 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:21.364393    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:13:26.928889 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:26.928849    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" containerID="cri-o://32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8" gracePeriod=45
Oct 31 01:13:27.092839 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.092792    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main" containerID="cri-o://fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff" gracePeriod=45
Oct 31 01:13:27.496899 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.496865    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Oct 31 01:13:27.682811 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.682784    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:13:27.752160 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752138    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-model-cache\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.752319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752171    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b945160-2367-4c3d-be09-24c8ba425be6-kube-api-access-2s8m7\" (UniqueName: \"kubernetes.io/projected/0b945160-2367-4c3d-be09-24c8ba425be6-kube-api-access-2s8m7\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.752319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752190    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b945160-2367-4c3d-be09-24c8ba425be6-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b945160-2367-4c3d-be09-24c8ba425be6-tls-certs\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.752319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752251    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-tmp-dir\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.752319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752276    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-kserve-provision-location\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.752319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752310    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-dshm\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.752648 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752473    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-model-cache" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:27.752648 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.752636    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b945160-2367-4c3d-be09-24c8ba425be6-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b945160-2367-4c3d-be09-24c8ba425be6-cabundle-cert\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.753201 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.753171    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b945160-2367-4c3d-be09-24c8ba425be6-cabundle-cert" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:27.753303 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.753238    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-home\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-home\") pod \"0b945160-2367-4c3d-be09-24c8ba425be6\" (UID: \"0b945160-2367-4c3d-be09-24c8ba425be6\") "
Oct 31 01:13:27.753633 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.753615    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0b945160-2367-4c3d-be09-24c8ba425be6-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.753732 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.753638    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.754015 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.753991    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-home" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:27.754742 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.754718    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b945160-2367-4c3d-be09-24c8ba425be6-kube-api-access-2s8m7" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "kube-api-access-2s8m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:27.755146 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.755115    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-dshm" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:27.755884 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.755861    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b945160-2367-4c3d-be09-24c8ba425be6-tls-certs" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:13:27.766985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.766959    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-tmp-dir" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:27.781625 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.781599    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-kserve-provision-location" pod "0b945160-2367-4c3d-be09-24c8ba425be6" (UID: "0b945160-2367-4c3d-be09-24c8ba425be6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:13:27.854953 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.854924    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.854953 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.854949    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.855100 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.854965    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.855100 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.854978    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b945160-2367-4c3d-be09-24c8ba425be6-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.855100 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.854989    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s8m7\" (UniqueName: \"kubernetes.io/projected/0b945160-2367-4c3d-be09-24c8ba425be6-kube-api-access-2s8m7\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.855100 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.855003    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b945160-2367-4c3d-be09-24c8ba425be6-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:27.981111 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.981075    4888 generic.go:356] "Generic (PLEG): container finished" podID="0b945160-2367-4c3d-be09-24c8ba425be6" containerID="32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8" exitCode=0
Oct 31 01:13:27.981551 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.981121    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" event={"ID":"0b945160-2367-4c3d-be09-24c8ba425be6","Type":"ContainerDied","Data":"32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8"}
Oct 31 01:13:27.981551 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.981144    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"
Oct 31 01:13:27.981551 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.981186    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9" event={"ID":"0b945160-2367-4c3d-be09-24c8ba425be6","Type":"ContainerDied","Data":"95fd52692bc8531dd75391710f2628bdcd7b68bd0e613ee1cd5c5bdd651203bd"}
Oct 31 01:13:27.981551 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.981207    4888 scope.go:122] "RemoveContainer" containerID="32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8"
Oct 31 01:13:27.990166 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.990145    4888 scope.go:122] "RemoveContainer" containerID="a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d"
Oct 31 01:13:27.997337 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.997318    4888 scope.go:122] "RemoveContainer" containerID="32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8"
Oct 31 01:13:27.997678 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:13:27.997658    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8\": container with ID starting with 32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8 not found: ID does not exist" containerID="32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8"
Oct 31 01:13:27.997756 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.997685    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8"} err="failed to get container status \"32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8\": rpc error: code = NotFound desc = could not find container \"32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8\": container with ID starting with 32caaa1feffbe4d822c249061f4c21c55dcd7d4ed10793ae2b3103bd58c13ac8 not found: ID does not exist"
Oct 31 01:13:27.997756 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.997703    4888 scope.go:122] "RemoveContainer" containerID="a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d"
Oct 31 01:13:27.997918 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:13:27.997904    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d\": container with ID starting with a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d not found: ID does not exist" containerID="a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d"
Oct 31 01:13:27.997970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:27.997921    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d"} err="failed to get container status \"a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d\": rpc error: code = NotFound desc = could not find container \"a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d\": container with ID starting with a60718aad8a3b92f7913697ab86c912a79a77768d5333837790a4299e01a387d not found: ID does not exist"
Oct 31 01:13:28.145139 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:28.145066    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"]
Oct 31 01:13:28.190936 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:28.190900    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-5678d64c4-8jdk9"]
Oct 31 01:13:29.554187 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:29.554156    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" path="/var/lib/kubelet/pods/0b945160-2367-4c3d-be09-24c8ba425be6/volumes"
Oct 31 01:13:31.364275 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:31.364238    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:13:41.363930 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:41.363892    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:13:41.364333 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:41.363988    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:13:48.552260 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:13:48.552224    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:13:51.364106 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:51.364012    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:13:57.334900 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:57.334873    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:13:57.401615 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:57.401553    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eddb732-84a7-4108-917a-7cdd29e6680b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddb732-84a7-4108-917a-7cdd29e6680b-tls-certs\") pod \"4eddb732-84a7-4108-917a-7cdd29e6680b\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") "
Oct 31 01:13:57.401615 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:57.401605    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eddb732-84a7-4108-917a-7cdd29e6680b-kube-api-access-p2w2z\" (UniqueName: \"kubernetes.io/projected/4eddb732-84a7-4108-917a-7cdd29e6680b-kube-api-access-p2w2z\") pod \"4eddb732-84a7-4108-917a-7cdd29e6680b\" (UID: \"4eddb732-84a7-4108-917a-7cdd29e6680b\") "
Oct 31 01:13:57.403697 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:57.403666    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eddb732-84a7-4108-917a-7cdd29e6680b-tls-certs" pod "4eddb732-84a7-4108-917a-7cdd29e6680b" (UID: "4eddb732-84a7-4108-917a-7cdd29e6680b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:13:57.403697 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:57.403689    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eddb732-84a7-4108-917a-7cdd29e6680b-kube-api-access-p2w2z" pod "4eddb732-84a7-4108-917a-7cdd29e6680b" (UID: "4eddb732-84a7-4108-917a-7cdd29e6680b"). InnerVolumeSpecName "kube-api-access-p2w2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:57.502813 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:57.502784    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4eddb732-84a7-4108-917a-7cdd29e6680b-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:57.502813 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:57.502811    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2w2z\" (UniqueName: \"kubernetes.io/projected/4eddb732-84a7-4108-917a-7cdd29e6680b-kube-api-access-p2w2z\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:13:58.121314 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.121280    4888 generic.go:356] "Generic (PLEG): container finished" podID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerID="fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff" exitCode=0
Oct 31 01:13:58.121523 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.121326    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"
Oct 31 01:13:58.121523 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.121385    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" event={"ID":"4eddb732-84a7-4108-917a-7cdd29e6680b","Type":"ContainerDied","Data":"fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff"}
Oct 31 01:13:58.121523 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.121438    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w" event={"ID":"4eddb732-84a7-4108-917a-7cdd29e6680b","Type":"ContainerDied","Data":"57f61b6e4608f827a297d1b7163f2c71b901e739fd164cb99f6ea23cf3fa895e"}
Oct 31 01:13:58.121523 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.121465    4888 scope.go:122] "RemoveContainer" containerID="fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff"
Oct 31 01:13:58.129621 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.129605    4888 scope.go:122] "RemoveContainer" containerID="fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff"
Oct 31 01:13:58.129879 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:13:58.129860    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff\": container with ID starting with fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff not found: ID does not exist" containerID="fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff"
Oct 31 01:13:58.129952 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.129886    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff"} err="failed to get container status \"fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff\": rpc error: code = NotFound desc = could not find container \"fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff\": container with ID starting with fe428b0ee09659580e7e4d9eebcf6605d89e8f9cefc0ecc2dbbced42fcfd19ff not found: ID does not exist"
Oct 31 01:13:58.281336 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.281308    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"]
Oct 31 01:13:58.328135 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:58.328102    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-5n62w"]
Oct 31 01:13:59.553242 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:13:59.553208    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" path="/var/lib/kubelet/pods/4eddb732-84a7-4108-917a-7cdd29e6680b/volumes"
Oct 31 01:14:18.552105 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:14:18.552069    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:14:41.779049 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779014    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"]
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779327    4888 cpu_manager.go:414] "containerMap: removing container" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main"
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779337    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main"
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779372    4888 cpu_manager.go:414] "containerMap: removing container" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main"
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779377    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main"
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779395    4888 cpu_manager.go:414] "containerMap: removing container" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="storage-initializer"
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779401    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="storage-initializer"
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779460    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b945160-2367-4c3d-be09-24c8ba425be6" containerName="main"
Oct 31 01:14:41.779562 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.779470    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eddb732-84a7-4108-917a-7cdd29e6680b" containerName="main"
Oct 31 01:14:41.784132 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.784103    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.835577 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.835546    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 31 01:14:41.865204 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865175    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.865320 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865229    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64406bc0-ff0b-43da-969c-22aeecaa032f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.865320 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865289    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkbsb\" (UniqueName: \"kubernetes.io/projected/64406bc0-ff0b-43da-969c-22aeecaa032f-kube-api-access-gkbsb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.865320 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865316    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64406bc0-ff0b-43da-969c-22aeecaa032f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.865455 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865334    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.865455 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865374    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.865518 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865456    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.865518 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.865481    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.882242 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.882214    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-vbflt\""
Oct 31 01:14:41.882385 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.882312    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-mrwg5\""
Oct 31 01:14:41.882385 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.882337    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:14:41.882385 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.882332    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 31 01:14:41.882538 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.882389    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 31 01:14:41.882538 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.882427    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 31 01:14:41.898628 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.898606    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"]
Oct 31 01:14:41.965927 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.965896    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64406bc0-ff0b-43da-969c-22aeecaa032f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966110 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.965938    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkbsb\" (UniqueName: \"kubernetes.io/projected/64406bc0-ff0b-43da-969c-22aeecaa032f-kube-api-access-gkbsb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966110 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.965972    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64406bc0-ff0b-43da-969c-22aeecaa032f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966110 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.965992    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966110 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966016    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966340 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966119    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966340 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966149    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966340 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966200    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966566 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966461    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966566 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966536    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966846 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966814    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966846 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966837    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.966979 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.966956    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64406bc0-ff0b-43da-969c-22aeecaa032f-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.968951 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.968924    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:41.969073 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:41.969054    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64406bc0-ff0b-43da-969c-22aeecaa032f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:42.026087 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.026060    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkbsb\" (UniqueName: \"kubernetes.io/projected/64406bc0-ff0b-43da-969c-22aeecaa032f-kube-api-access-gkbsb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:42.101875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.101785    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:42.440104 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.440021    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"]
Oct 31 01:14:42.443775 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.443748    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:42.472763 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.472739    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkgh\" (UniqueName: \"kubernetes.io/projected/00fb5701-5c53-4c1a-82b7-b520d4144a0b-kube-api-access-kkkgh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:42.472897 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.472770    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb5701-5c53-4c1a-82b7-b520d4144a0b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:42.495094 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.495071    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-8wpgg\""
Oct 31 01:14:42.506803 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.506781    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"]
Oct 31 01:14:42.536512 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.536487    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"]
Oct 31 01:14:42.544263 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:14:42.544237    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64406bc0_ff0b_43da_969c_22aeecaa032f.slice/crio-4b4dfdcf2a7d2661a01c62a9d9a630a77e8cf290b30a6ba4039fa2262f25fabe WatchSource:0}: Error finding container 4b4dfdcf2a7d2661a01c62a9d9a630a77e8cf290b30a6ba4039fa2262f25fabe: Status 404 returned error can't find the container with id 4b4dfdcf2a7d2661a01c62a9d9a630a77e8cf290b30a6ba4039fa2262f25fabe
Oct 31 01:14:42.546261 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.546246    4888 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:14:42.573699 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.573669    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkgh\" (UniqueName: \"kubernetes.io/projected/00fb5701-5c53-4c1a-82b7-b520d4144a0b-kube-api-access-kkkgh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:42.573833 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.573702    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb5701-5c53-4c1a-82b7-b520d4144a0b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:42.576308 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.576289    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb5701-5c53-4c1a-82b7-b520d4144a0b-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:42.636341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.636311    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkgh\" (UniqueName: \"kubernetes.io/projected/00fb5701-5c53-4c1a-82b7-b520d4144a0b-kube-api-access-kkkgh\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:42.756309 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:42.756267    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:43.132646 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:43.132564    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"]
Oct 31 01:14:43.140489 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:14:43.140462    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00fb5701_5c53_4c1a_82b7_b520d4144a0b.slice/crio-180be0c9a6bbc8e1affd0866496f3b5402ceaf240d9184042c5c02fe502e53c0 WatchSource:0}: Error finding container 180be0c9a6bbc8e1affd0866496f3b5402ceaf240d9184042c5c02fe502e53c0: Status 404 returned error can't find the container with id 180be0c9a6bbc8e1affd0866496f3b5402ceaf240d9184042c5c02fe502e53c0
Oct 31 01:14:43.326075 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:43.326038    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" event={"ID":"00fb5701-5c53-4c1a-82b7-b520d4144a0b","Type":"ContainerStarted","Data":"69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003"}
Oct 31 01:14:43.326272 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:43.326093    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" event={"ID":"00fb5701-5c53-4c1a-82b7-b520d4144a0b","Type":"ContainerStarted","Data":"180be0c9a6bbc8e1affd0866496f3b5402ceaf240d9184042c5c02fe502e53c0"}
Oct 31 01:14:43.326272 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:43.326124    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:14:43.327247 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:43.327222    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerStarted","Data":"4b4dfdcf2a7d2661a01c62a9d9a630a77e8cf290b30a6ba4039fa2262f25fabe"}
Oct 31 01:14:43.440255 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:43.440153    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" podStartSLOduration=1.440140523 podStartE2EDuration="1.440140523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:43.438547038 +0000 UTC m=+2691.993434475" watchObservedRunningTime="2026-10-31 01:14:43.440140523 +0000 UTC m=+2691.995027983"
Oct 31 01:14:47.351843 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:47.351809    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerStarted","Data":"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"}
Oct 31 01:14:47.352339 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:47.351913    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:48.359771 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:48.359738    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerStarted","Data":"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"}
Oct 31 01:14:48.554994 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:14:48.554952    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:14:50.370691 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:50.370658    4888 generic.go:356] "Generic (PLEG): container finished" podID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerID="71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546" exitCode=0
Oct 31 01:14:50.370691 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:50.370694    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerDied","Data":"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"}
Oct 31 01:14:51.377730 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:51.377693    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerStarted","Data":"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"}
Oct 31 01:14:51.593498 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:51.593451    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podStartSLOduration=4.768692904 podStartE2EDuration="10.593436289s" totalImagesPullingTime="3.766990037s" totalInitContainerRuntime="2.057753348s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:14:51.545256539 +0000 UTC m=+2700.100143976" watchObservedRunningTime="2026-10-31 01:14:51.593436289 +0000 UTC m=+2700.148323723"
Oct 31 01:14:51.837739 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:51.837711    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:14:51.847285 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:51.847255    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:14:52.102980 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:52.102891    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:52.102980 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:52.102980    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:14:52.104095 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:14:52.104069    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:15:02.102203 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:02.102148    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:15:02.394157 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:02.394073    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:15:12.102664 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:12.102623    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:15:14.336029 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:14.335995    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:15:18.552979 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:15:18.552943    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:15:22.102274 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:22.102173    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:15:32.103045 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:32.103003    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:15:42.102401 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:42.102329    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:15:46.943337 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:46.943298    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"]
Oct 31 01:15:46.943832 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:46.943512    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" containerID="cri-o://02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42" gracePeriod=30
Oct 31 01:15:48.306898 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:48.306863    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-dcvp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:15:48.307386 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:48.306915    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:15:48.552949 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:15:48.552913    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:15:52.103206 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.103149    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:15:52.194911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.194885    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:15:52.291188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291105    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-socket\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291143    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-envoy\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291166    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-kube-api-access-hwfq4\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-kube-api-access-hwfq4\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291192    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-certs\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291219    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-podinfo\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291273    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14c415a-7a4a-402e-b3f4-f5531312db7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14c415a-7a4a-402e-b3f4-f5531312db7e-istiod-ca-cert\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291317    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-credential-socket\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291342    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-data\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291428    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-token\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-token\") pod \"f14c415a-7a4a-402e-b3f4-f5531312db7e\" (UID: \"f14c415a-7a4a-402e-b3f4-f5531312db7e\") "
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291446    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-socket" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:52.291547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291528    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-certs" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:52.291977 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291614    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-credential-socket" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:52.291977 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291731    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.291977 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291747    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-data" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:52.291977 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291753    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.291977 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.291785    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.292417 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.292286    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14c415a-7a4a-402e-b3f4-f5531312db7e-istiod-ca-cert" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:52.293674 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.293644    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-podinfo" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:15:52.293795 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.293713    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-envoy" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:15:52.294094 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.294068    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-kube-api-access-hwfq4" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "kube-api-access-hwfq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:52.294094 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.294066    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-token" pod "f14c415a-7a4a-402e-b3f4-f5531312db7e" (UID: "f14c415a-7a4a-402e-b3f4-f5531312db7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:52.392266 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.392232    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.392448 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.392272    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwfq4\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-kube-api-access-hwfq4\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.392448 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.392290    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.392448 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.392305    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f14c415a-7a4a-402e-b3f4-f5531312db7e-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.392448 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.392318    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.392448 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.392332    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f14c415a-7a4a-402e-b3f4-f5531312db7e-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:15:52.694062 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.693972    4888 generic.go:356] "Generic (PLEG): container finished" podID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerID="02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42" exitCode=0
Oct 31 01:15:52.694062 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.694027    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"
Oct 31 01:15:52.694308 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.694061    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" event={"ID":"f14c415a-7a4a-402e-b3f4-f5531312db7e","Type":"ContainerDied","Data":"02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42"}
Oct 31 01:15:52.694308 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.694126    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9" event={"ID":"f14c415a-7a4a-402e-b3f4-f5531312db7e","Type":"ContainerDied","Data":"981bb20459413ade34be06c88e16e3320d64fadd6707d78fd481afd1a985a33d"}
Oct 31 01:15:52.694308 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.694158    4888 scope.go:122] "RemoveContainer" containerID="02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42"
Oct 31 01:15:52.705384 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.705362    4888 scope.go:122] "RemoveContainer" containerID="02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42"
Oct 31 01:15:52.705680 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:15:52.705663    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42\": container with ID starting with 02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42 not found: ID does not exist" containerID="02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42"
Oct 31 01:15:52.705736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.705691    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42"} err="failed to get container status \"02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42\": rpc error: code = NotFound desc = could not find container \"02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42\": container with ID starting with 02cca1d5c1ea3969d0c74cc3232282d17fc52bd13ff2cdd35d14faa71a8a7e42 not found: ID does not exist"
Oct 31 01:15:52.854204 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.854171    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"]
Oct 31 01:15:52.900845 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:52.900817    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-dcvp9"]
Oct 31 01:15:53.553609 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:15:53.553567    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" path="/var/lib/kubelet/pods/f14c415a-7a4a-402e-b3f4-f5531312db7e/volumes"
Oct 31 01:16:02.102374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:02.102293    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:16:12.103092 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:12.103035    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:16:18.552422 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:16:18.552383    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:16:22.102210 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:22.102169    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:16:32.102835 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:32.102765    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8001/health\": dial tcp 10.134.0.65:8001: connect: connection refused"
Oct 31 01:16:42.112697 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:42.112666    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:16:42.123879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:42.123855    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:16:47.444705 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:47.444672    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"]
Oct 31 01:16:47.445103 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:47.444855    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" containerID="cri-o://ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba" gracePeriod=30
Oct 31 01:16:48.351703 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:48.351671    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-c7ln8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:16:48.351868 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:48.351737    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:16:48.551993 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:16:48.551956    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:16:52.692506 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.692480    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:16:52.781482 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781389    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-envoy\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781482 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781429    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-credential-socket\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781482 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781460    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-podinfo\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781482 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781484    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-socket\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781820 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781509    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-data\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781820 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781537    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b5765b-2435-4523-a9cb-8a60354078ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b5765b-2435-4523-a9cb-8a60354078ec-istiod-ca-cert\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781820 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781756    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-credential-socket" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:52.781970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781838    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-data" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:52.781970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781849    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-socket" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:52.781970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781874    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-certs\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781945    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-token\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-token\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.781970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781946    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b5765b-2435-4523-a9cb-8a60354078ec-istiod-ca-cert" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:16:52.782217 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.781976    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-kube-api-access-phcj5\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-kube-api-access-phcj5\") pod \"b6b5765b-2435-4523-a9cb-8a60354078ec\" (UID: \"b6b5765b-2435-4523-a9cb-8a60354078ec\") "
Oct 31 01:16:52.782217 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.782051    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-certs" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:52.782319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.782231    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.782319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.782252    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.782319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.782266    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.782319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.782279    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b5765b-2435-4523-a9cb-8a60354078ec-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.782319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.782293    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.783950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.783920    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-envoy" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:16:52.784102 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.783987    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-podinfo" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:16:52.784396 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.784371    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-kube-api-access-phcj5" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "kube-api-access-phcj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:52.784396 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.784390    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-token" pod "b6b5765b-2435-4523-a9cb-8a60354078ec" (UID: "b6b5765b-2435-4523-a9cb-8a60354078ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:16:52.882869 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.882832    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.882869 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.882861    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phcj5\" (UniqueName: \"kubernetes.io/projected/b6b5765b-2435-4523-a9cb-8a60354078ec-kube-api-access-phcj5\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.882869 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.882871    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.883170 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.882881    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b5765b-2435-4523-a9cb-8a60354078ec-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:16:52.997908 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.997872    4888 generic.go:356] "Generic (PLEG): container finished" podID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerID="ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba" exitCode=0
Oct 31 01:16:52.998081 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.997946    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"
Oct 31 01:16:52.998081 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.997968    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" event={"ID":"b6b5765b-2435-4523-a9cb-8a60354078ec","Type":"ContainerDied","Data":"ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba"}
Oct 31 01:16:52.998081 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.998024    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8" event={"ID":"b6b5765b-2435-4523-a9cb-8a60354078ec","Type":"ContainerDied","Data":"8e103f1b8373f8f3e5fe4fbdf66429a97b159a79c5ba4f0e7317df3b4664e0eb"}
Oct 31 01:16:52.998081 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:52.998046    4888 scope.go:122] "RemoveContainer" containerID="ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba"
Oct 31 01:16:53.006949 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:53.006929    4888 scope.go:122] "RemoveContainer" containerID="ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba"
Oct 31 01:16:53.007299 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:16:53.007232    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba\": container with ID starting with ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba not found: ID does not exist" containerID="ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba"
Oct 31 01:16:53.007299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:53.007272    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba"} err="failed to get container status \"ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba\": rpc error: code = NotFound desc = could not find container \"ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba\": container with ID starting with ba1693e9697a8781a027a8cce97acb3d83f8bad59d3eeffca765f60e366810ba not found: ID does not exist"
Oct 31 01:16:53.158492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:53.158415    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"]
Oct 31 01:16:53.165394 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:53.165366    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-c7ln8"]
Oct 31 01:16:53.555536 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:53.555500    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" path="/var/lib/kubelet/pods/b6b5765b-2435-4523-a9cb-8a60354078ec/volumes"
Oct 31 01:16:55.947799 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:55.947765    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"]
Oct 31 01:16:55.957456 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:16:55.957429    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"]
Oct 31 01:17:04.335528 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:04.335491    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:17:10.968236 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:10.968178    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main" containerID="cri-o://69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003" gracePeriod=45
Oct 31 01:17:10.984449 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:10.984410    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main" containerID="cri-o://f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012" gracePeriod=45
Oct 31 01:17:11.504828 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.504802    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="llm-d-routing-sidecar" containerID="cri-o://32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544" gracePeriod=45
Oct 31 01:17:11.741613 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.741588    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:17:11.835319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835293    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64406bc0-ff0b-43da-969c-22aeecaa032f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64406bc0-ff0b-43da-969c-22aeecaa032f-cabundle-cert\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835319 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835326    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-kserve-provision-location\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835576 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835374    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64406bc0-ff0b-43da-969c-22aeecaa032f-kube-api-access-gkbsb\" (UniqueName: \"kubernetes.io/projected/64406bc0-ff0b-43da-969c-22aeecaa032f-kube-api-access-gkbsb\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835576 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835398    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-home\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-home\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835576 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835547    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-tmp-dir\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835798 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835678    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-model-cache\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835798 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835692    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64406bc0-ff0b-43da-969c-22aeecaa032f-cabundle-cert" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:17:11.835798 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835708    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-dshm\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835798 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835744    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64406bc0-ff0b-43da-969c-22aeecaa032f-tls-certs\" (UniqueName: \"kubernetes.io/secret/64406bc0-ff0b-43da-969c-22aeecaa032f-tls-certs\") pod \"64406bc0-ff0b-43da-969c-22aeecaa032f\" (UID: \"64406bc0-ff0b-43da-969c-22aeecaa032f\") "
Oct 31 01:17:11.835996 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.835959    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/64406bc0-ff0b-43da-969c-22aeecaa032f-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.836066 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.836045    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-model-cache" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.836161 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.836134    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-home" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.837877 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.837850    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64406bc0-ff0b-43da-969c-22aeecaa032f-kube-api-access-gkbsb" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "kube-api-access-gkbsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:11.837996 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.837972    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-dshm" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.838052 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.838031    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64406bc0-ff0b-43da-969c-22aeecaa032f-tls-certs" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:11.854243 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.854209    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-tmp-dir" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.864396 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.864368    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-kserve-provision-location" pod "64406bc0-ff0b-43da-969c-22aeecaa032f" (UID: "64406bc0-ff0b-43da-969c-22aeecaa032f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:17:11.936653 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.936624    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.936653 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.936647    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.936653 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.936657    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64406bc0-ff0b-43da-969c-22aeecaa032f-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.936889 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.936667    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.936889 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.936676    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkbsb\" (UniqueName: \"kubernetes.io/projected/64406bc0-ff0b-43da-969c-22aeecaa032f-kube-api-access-gkbsb\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.936889 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.936684    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:11.936889 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:11.936692    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64406bc0-ff0b-43da-969c-22aeecaa032f-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:12.093026 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.092935    4888 generic.go:356] "Generic (PLEG): container finished" podID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerID="f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012" exitCode=0
Oct 31 01:17:12.093026 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.092965    4888 generic.go:356] "Generic (PLEG): container finished" podID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerID="32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544" exitCode=0
Oct 31 01:17:12.093026 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.093003    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"
Oct 31 01:17:12.093568 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.093022    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerDied","Data":"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"}
Oct 31 01:17:12.093568 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.093084    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerDied","Data":"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"}
Oct 31 01:17:12.093568 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.093101    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4" event={"ID":"64406bc0-ff0b-43da-969c-22aeecaa032f","Type":"ContainerDied","Data":"4b4dfdcf2a7d2661a01c62a9d9a630a77e8cf290b30a6ba4039fa2262f25fabe"}
Oct 31 01:17:12.093568 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.093123    4888 scope.go:122] "RemoveContainer" containerID="f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"
Oct 31 01:17:12.102013 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.101996    4888 scope.go:122] "RemoveContainer" containerID="71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"
Oct 31 01:17:12.109638 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.109621    4888 scope.go:122] "RemoveContainer" containerID="32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"
Oct 31 01:17:12.116716 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.116699    4888 scope.go:122] "RemoveContainer" containerID="f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"
Oct 31 01:17:12.116982 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:17:12.116952    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012\": container with ID starting with f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012 not found: ID does not exist" containerID="f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"
Oct 31 01:17:12.117047 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.116983    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"} err="failed to get container status \"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012\": rpc error: code = NotFound desc = could not find container \"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012\": container with ID starting with f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012 not found: ID does not exist"
Oct 31 01:17:12.117047 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117001    4888 scope.go:122] "RemoveContainer" containerID="71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"
Oct 31 01:17:12.117253 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:17:12.117237    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546\": container with ID starting with 71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546 not found: ID does not exist" containerID="71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"
Oct 31 01:17:12.117298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117260    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"} err="failed to get container status \"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546\": rpc error: code = NotFound desc = could not find container \"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546\": container with ID starting with 71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546 not found: ID does not exist"
Oct 31 01:17:12.117298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117281    4888 scope.go:122] "RemoveContainer" containerID="32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"
Oct 31 01:17:12.117523 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:17:12.117508    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544\": container with ID starting with 32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544 not found: ID does not exist" containerID="32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"
Oct 31 01:17:12.117570 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117527    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"} err="failed to get container status \"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544\": rpc error: code = NotFound desc = could not find container \"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544\": container with ID starting with 32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544 not found: ID does not exist"
Oct 31 01:17:12.117570 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117538    4888 scope.go:122] "RemoveContainer" containerID="f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"
Oct 31 01:17:12.117762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117746    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012"} err="failed to get container status \"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012\": rpc error: code = NotFound desc = could not find container \"f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012\": container with ID starting with f72311ec78637ac1d7ff294e8c929c4eb767f509c8db646ebc662aa993b63012 not found: ID does not exist"
Oct 31 01:17:12.117811 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117763    4888 scope.go:122] "RemoveContainer" containerID="71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"
Oct 31 01:17:12.117959 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117943    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546"} err="failed to get container status \"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546\": rpc error: code = NotFound desc = could not find container \"71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546\": container with ID starting with 71ae9ab47bc1f873231cf9e64e4814377510b7caff09e923e67132601d65f546 not found: ID does not exist"
Oct 31 01:17:12.117959 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.117958    4888 scope.go:122] "RemoveContainer" containerID="32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"
Oct 31 01:17:12.118155 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.118138    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544"} err="failed to get container status \"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544\": rpc error: code = NotFound desc = could not find container \"32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544\": container with ID starting with 32dc5401507f3fe90b6c1a72c7643a5a5c0e21967d03a833b19a8e8dbb325544 not found: ID does not exist"
Oct 31 01:17:12.253341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.253311    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"]
Oct 31 01:17:12.300821 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:12.300792    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-5bf4f78ccd6d9v4"]
Oct 31 01:17:13.553631 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:13.553593    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" path="/var/lib/kubelet/pods/64406bc0-ff0b-43da-969c-22aeecaa032f/volumes"
Oct 31 01:17:14.335760 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:14.335719    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:17:18.551234 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:17:18.551197    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:17:24.335673 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:24.335638    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:17:24.336249 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:24.335764    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:17:34.335227 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:34.335184    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:17:41.214593 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.214569    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:17:41.228193 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.228133    4888 generic.go:356] "Generic (PLEG): container finished" podID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerID="69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003" exitCode=0
Oct 31 01:17:41.228193 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.228171    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"
Oct 31 01:17:41.228392 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.228213    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" event={"ID":"00fb5701-5c53-4c1a-82b7-b520d4144a0b","Type":"ContainerDied","Data":"69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003"}
Oct 31 01:17:41.228392 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.228278    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m" event={"ID":"00fb5701-5c53-4c1a-82b7-b520d4144a0b","Type":"ContainerDied","Data":"180be0c9a6bbc8e1affd0866496f3b5402ceaf240d9184042c5c02fe502e53c0"}
Oct 31 01:17:41.228392 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.228306    4888 scope.go:122] "RemoveContainer" containerID="69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003"
Oct 31 01:17:41.236381 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.236342    4888 scope.go:122] "RemoveContainer" containerID="69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003"
Oct 31 01:17:41.236655 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:17:41.236637    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003\": container with ID starting with 69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003 not found: ID does not exist" containerID="69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003"
Oct 31 01:17:41.236702 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.236664    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003"} err="failed to get container status \"69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003\": rpc error: code = NotFound desc = could not find container \"69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003\": container with ID starting with 69edb0b2c354391b5df28ee019d69db9966df80c3960b6067a6c6f679e44d003 not found: ID does not exist"
Oct 31 01:17:41.372488 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.372457    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00fb5701-5c53-4c1a-82b7-b520d4144a0b-kube-api-access-kkkgh\" (UniqueName: \"kubernetes.io/projected/00fb5701-5c53-4c1a-82b7-b520d4144a0b-kube-api-access-kkkgh\") pod \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") "
Oct 31 01:17:41.372674 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.372534    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00fb5701-5c53-4c1a-82b7-b520d4144a0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb5701-5c53-4c1a-82b7-b520d4144a0b-tls-certs\") pod \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\" (UID: \"00fb5701-5c53-4c1a-82b7-b520d4144a0b\") "
Oct 31 01:17:41.374745 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.374720    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00fb5701-5c53-4c1a-82b7-b520d4144a0b-kube-api-access-kkkgh" pod "00fb5701-5c53-4c1a-82b7-b520d4144a0b" (UID: "00fb5701-5c53-4c1a-82b7-b520d4144a0b"). InnerVolumeSpecName "kube-api-access-kkkgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:17:41.374852 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.374759    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00fb5701-5c53-4c1a-82b7-b520d4144a0b-tls-certs" pod "00fb5701-5c53-4c1a-82b7-b520d4144a0b" (UID: "00fb5701-5c53-4c1a-82b7-b520d4144a0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:17:41.473707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.473676    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkkgh\" (UniqueName: \"kubernetes.io/projected/00fb5701-5c53-4c1a-82b7-b520d4144a0b-kube-api-access-kkkgh\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:41.473707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.473701    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00fb5701-5c53-4c1a-82b7-b520d4144a0b-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:17:41.645127 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.645093    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"]
Oct 31 01:17:41.650434 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:41.650404    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheqdm9m"]
Oct 31 01:17:43.553414 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:43.553381    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" path="/var/lib/kubelet/pods/00fb5701-5c53-4c1a-82b7-b520d4144a0b/volumes"
Oct 31 01:17:48.553157 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:17:48.553120    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:17:51.027217 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027186    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"]
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027509    4888 cpu_manager.go:414] "containerMap: removing container" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="llm-d-routing-sidecar"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027520    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="llm-d-routing-sidecar"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027535    4888 cpu_manager.go:414] "containerMap: removing container" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027540    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027554    4888 cpu_manager.go:414] "containerMap: removing container" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027559    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027568    4888 cpu_manager.go:414] "containerMap: removing container" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027572    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027582    4888 cpu_manager.go:414] "containerMap: removing container" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="storage-initializer"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027586    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="storage-initializer"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027599    4888 cpu_manager.go:414] "containerMap: removing container" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main"
Oct 31 01:17:51.027651 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027604    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main"
Oct 31 01:17:51.028001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027678    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b5765b-2435-4523-a9cb-8a60354078ec" containerName="istio-proxy"
Oct 31 01:17:51.028001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027687    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="main"
Oct 31 01:17:51.028001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027695    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="00fb5701-5c53-4c1a-82b7-b520d4144a0b" containerName="main"
Oct 31 01:17:51.028001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027701    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="64406bc0-ff0b-43da-969c-22aeecaa032f" containerName="llm-d-routing-sidecar"
Oct 31 01:17:51.028001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.027707    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14c415a-7a4a-402e-b3f4-f5531312db7e" containerName="istio-proxy"
Oct 31 01:17:51.032167 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.032145    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.083368 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.083319    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 31 01:17:51.083557 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.083538    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 31 01:17:51.083604 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.083553    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-84dhk\""
Oct 31 01:17:51.083638 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.083624    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:17:51.130945 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.130918    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 31 01:17:51.130945 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.130921    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 31 01:17:51.131099 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.130959    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-wxft4\""
Oct 31 01:17:51.146478 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.146451    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"]
Oct 31 01:17:51.154100 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154077    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/815cdf55-5244-427c-a68f-0f243e3990fa-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.154208 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154157    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-dshm\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.154272 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154253    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.154334 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154316    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815cdf55-5244-427c-a68f-0f243e3990fa-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.154458 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154436    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.154518 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154501    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52qb8\" (UniqueName: \"kubernetes.io/projected/815cdf55-5244-427c-a68f-0f243e3990fa-kube-api-access-52qb8\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.154554 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154534    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-home\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.154598 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.154558    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-model-cache\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255226 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255185    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-dshm\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255226 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255225    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255479 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255394    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815cdf55-5244-427c-a68f-0f243e3990fa-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255479 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255431    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255580 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255484    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52qb8\" (UniqueName: \"kubernetes.io/projected/815cdf55-5244-427c-a68f-0f243e3990fa-kube-api-access-52qb8\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255580 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255511    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-home\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255580 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255534    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-model-cache\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255737 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255598    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/815cdf55-5244-427c-a68f-0f243e3990fa-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255737 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255647    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255845 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255831    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.255924 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.255902    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-home\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.256159 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.256139    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-model-cache\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.256496 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.256474    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/815cdf55-5244-427c-a68f-0f243e3990fa-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.257832 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.257810    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-dshm\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.258097 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.258074    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815cdf55-5244-427c-a68f-0f243e3990fa-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.317143 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.317075    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52qb8\" (UniqueName: \"kubernetes.io/projected/815cdf55-5244-427c-a68f-0f243e3990fa-kube-api-access-52qb8\") pod \"custom-route-timeout-pd-test-kserve-64975dd687-qx94s\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.347816 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.347778    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:51.778666 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:51.778636    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"]
Oct 31 01:17:51.785925 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:17:51.785896    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815cdf55_5244_427c_a68f_0f243e3990fa.slice/crio-1df873685c5f354c6aca3c9c810d2fe5ec4dc1199021dc618b6e38c2a290fdde WatchSource:0}: Error finding container 1df873685c5f354c6aca3c9c810d2fe5ec4dc1199021dc618b6e38c2a290fdde: Status 404 returned error can't find the container with id 1df873685c5f354c6aca3c9c810d2fe5ec4dc1199021dc618b6e38c2a290fdde
Oct 31 01:17:52.282911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:52.282824    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerStarted","Data":"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"}
Oct 31 01:17:52.282911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:52.282880    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerStarted","Data":"1df873685c5f354c6aca3c9c810d2fe5ec4dc1199021dc618b6e38c2a290fdde"}
Oct 31 01:17:52.282911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:52.282909    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:17:52.332238 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:52.332199    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 31 01:17:53.288896 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:53.288860    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerStarted","Data":"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"}
Oct 31 01:17:55.298934 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:55.298900    4888 generic.go:356] "Generic (PLEG): container finished" podID="815cdf55-5244-427c-a68f-0f243e3990fa" containerID="686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7" exitCode=0
Oct 31 01:17:55.299338 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:55.298941    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerDied","Data":"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"}
Oct 31 01:17:56.306894 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:56.306854    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerStarted","Data":"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"}
Oct 31 01:17:56.522913 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:17:56.522848    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podStartSLOduration=4.492912276 podStartE2EDuration="6.522831023s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.029918747s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:56.474685054 +0000 UTC m=+2885.029572490" watchObservedRunningTime="2026-10-31 01:17:56.522831023 +0000 UTC m=+2885.077718457"
Oct 31 01:18:01.348437 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:01.348381    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:18:01.348952 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:01.348464    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:18:01.349785 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:01.349752    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:18:01.361516 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:01.361491    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:18:11.348742 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:11.348690    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:18:18.552440 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:18:18.552398    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:18:21.348490 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:21.348397    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:18:31.348367 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:31.348313    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:18:41.348441 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:41.348398    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:18:48.553373 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:18:48.553311    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:18:51.348799 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:18:51.348733    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:19:01.348434 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:01.348387    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:19:11.348235 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:11.348180    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:19:18.551795 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:19:18.551752    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:19:21.348634 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:21.348580    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:19:31.348677 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:31.348634    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8001/health\": dial tcp 10.134.0.67:8001: connect: connection refused"
Oct 31 01:19:41.359019 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:41.358987    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:19:41.370017 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:41.369995    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:19:48.551946 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:19:48.551900    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:19:51.871001 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:51.870934    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:19:51.882773 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:51.882751    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:19:55.342795 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:19:55.342758    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"]
Oct 31 01:20:10.366892 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:10.366838    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main" containerID="cri-o://2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e" gracePeriod=45
Oct 31 01:20:10.984008 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:10.983977    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="llm-d-routing-sidecar" containerID="cri-o://affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7" gracePeriod=45
Oct 31 01:20:11.224576 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.224549    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:20:11.392954 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.392921    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-dshm\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393474 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393020    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-home\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-home\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393474 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393049    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-model-cache\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393474 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393097    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/815cdf55-5244-427c-a68f-0f243e3990fa-tls-certs\" (UniqueName: \"kubernetes.io/secret/815cdf55-5244-427c-a68f-0f243e3990fa-tls-certs\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393474 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393124    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-tmp-dir\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393474 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393153    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/815cdf55-5244-427c-a68f-0f243e3990fa-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/815cdf55-5244-427c-a68f-0f243e3990fa-cabundle-cert\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393474 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393190    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-kserve-provision-location\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393474 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393217    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/815cdf55-5244-427c-a68f-0f243e3990fa-kube-api-access-52qb8\" (UniqueName: \"kubernetes.io/projected/815cdf55-5244-427c-a68f-0f243e3990fa-kube-api-access-52qb8\") pod \"815cdf55-5244-427c-a68f-0f243e3990fa\" (UID: \"815cdf55-5244-427c-a68f-0f243e3990fa\") "
Oct 31 01:20:11.393865 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.393642    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-model-cache" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.394412 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.394369    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-home" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.395658 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.394577    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/815cdf55-5244-427c-a68f-0f243e3990fa-cabundle-cert" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:20:11.395658 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.394822    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.395658 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.394857    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.395658 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.394872    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/815cdf55-5244-427c-a68f-0f243e3990fa-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.396309 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.396282    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-dshm" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.396695 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.396640    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/815cdf55-5244-427c-a68f-0f243e3990fa-kube-api-access-52qb8" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "kube-api-access-52qb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:20:11.396776 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.396742    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/815cdf55-5244-427c-a68f-0f243e3990fa-tls-certs" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:20:11.408985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.408932    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-tmp-dir" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.426542 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.426516    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-kserve-provision-location" pod "815cdf55-5244-427c-a68f-0f243e3990fa" (UID: "815cdf55-5244-427c-a68f-0f243e3990fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:20:11.496099 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.496070    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.496099 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.496097    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/815cdf55-5244-427c-a68f-0f243e3990fa-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.496099 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.496106    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.496308 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.496117    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/815cdf55-5244-427c-a68f-0f243e3990fa-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.496308 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.496128    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52qb8\" (UniqueName: \"kubernetes.io/projected/815cdf55-5244-427c-a68f-0f243e3990fa-kube-api-access-52qb8\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:20:11.942705 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.942670    4888 generic.go:356] "Generic (PLEG): container finished" podID="815cdf55-5244-427c-a68f-0f243e3990fa" containerID="2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e" exitCode=0
Oct 31 01:20:11.942705 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.942696    4888 generic.go:356] "Generic (PLEG): container finished" podID="815cdf55-5244-427c-a68f-0f243e3990fa" containerID="affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7" exitCode=0
Oct 31 01:20:11.942958 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.942744    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"
Oct 31 01:20:11.942958 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.942751    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerDied","Data":"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"}
Oct 31 01:20:11.942958 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.942802    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerDied","Data":"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"}
Oct 31 01:20:11.942958 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.942821    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s" event={"ID":"815cdf55-5244-427c-a68f-0f243e3990fa","Type":"ContainerDied","Data":"1df873685c5f354c6aca3c9c810d2fe5ec4dc1199021dc618b6e38c2a290fdde"}
Oct 31 01:20:11.942958 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.942850    4888 scope.go:122] "RemoveContainer" containerID="2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"
Oct 31 01:20:11.951238 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.951222    4888 scope.go:122] "RemoveContainer" containerID="686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"
Oct 31 01:20:11.959028 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.959011    4888 scope.go:122] "RemoveContainer" containerID="affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"
Oct 31 01:20:11.966030 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.966013    4888 scope.go:122] "RemoveContainer" containerID="2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"
Oct 31 01:20:11.966315 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:20:11.966298    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e\": container with ID starting with 2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e not found: ID does not exist" containerID="2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"
Oct 31 01:20:11.966379 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.966329    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"} err="failed to get container status \"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e\": rpc error: code = NotFound desc = could not find container \"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e\": container with ID starting with 2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e not found: ID does not exist"
Oct 31 01:20:11.966379 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.966363    4888 scope.go:122] "RemoveContainer" containerID="686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"
Oct 31 01:20:11.966594 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:20:11.966577    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7\": container with ID starting with 686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7 not found: ID does not exist" containerID="686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"
Oct 31 01:20:11.966641 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.966600    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"} err="failed to get container status \"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7\": rpc error: code = NotFound desc = could not find container \"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7\": container with ID starting with 686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7 not found: ID does not exist"
Oct 31 01:20:11.966641 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.966614    4888 scope.go:122] "RemoveContainer" containerID="affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"
Oct 31 01:20:11.966873 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:20:11.966839    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7\": container with ID starting with affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7 not found: ID does not exist" containerID="affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"
Oct 31 01:20:11.966936 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.966880    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"} err="failed to get container status \"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7\": rpc error: code = NotFound desc = could not find container \"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7\": container with ID starting with affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7 not found: ID does not exist"
Oct 31 01:20:11.966936 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.966899    4888 scope.go:122] "RemoveContainer" containerID="2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"
Oct 31 01:20:11.967134 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.967118    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e"} err="failed to get container status \"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e\": rpc error: code = NotFound desc = could not find container \"2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e\": container with ID starting with 2f2b7324d1fe86560dc88958a4b5ac5430b3c4351ac6531cb2303afffe09f18e not found: ID does not exist"
Oct 31 01:20:11.967178 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.967134    4888 scope.go:122] "RemoveContainer" containerID="686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"
Oct 31 01:20:11.967336 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.967317    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7"} err="failed to get container status \"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7\": rpc error: code = NotFound desc = could not find container \"686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7\": container with ID starting with 686cb4f72f498d682d6e42bc364be82132978684defee6bf9821f64ef1517fc7 not found: ID does not exist"
Oct 31 01:20:11.967421 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.967338    4888 scope.go:122] "RemoveContainer" containerID="affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"
Oct 31 01:20:11.967566 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:11.967551    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7"} err="failed to get container status \"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7\": rpc error: code = NotFound desc = could not find container \"affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7\": container with ID starting with affcd8836be624e833d612101be95d8c7c679f64a2d980657b1b81a3fbcfb9d7 not found: ID does not exist"
Oct 31 01:20:12.149853 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:12.149816    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"]
Oct 31 01:20:12.199985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:12.199939    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-64975dd687-qx94s"]
Oct 31 01:20:13.552884 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:13.552852    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" path="/var/lib/kubelet/pods/815cdf55-5244-427c-a68f-0f243e3990fa/volumes"
Oct 31 01:20:18.551459 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:20:18.551427    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:20:45.466891 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.466857    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"]
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467304    4888 cpu_manager.go:414] "containerMap: removing container" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="llm-d-routing-sidecar"
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467322    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="llm-d-routing-sidecar"
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467367    4888 cpu_manager.go:414] "containerMap: removing container" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="storage-initializer"
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467376    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="storage-initializer"
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467387    4888 cpu_manager.go:414] "containerMap: removing container" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main"
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467392    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main"
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467455    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="main"
Oct 31 01:20:45.469341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.467467    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="815cdf55-5244-427c-a68f-0f243e3990fa" containerName="llm-d-routing-sidecar"
Oct 31 01:20:45.470152 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.470127    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.521492 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.521464    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 31 01:20:45.551322 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.551299    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Oct 31 01:20:45.551529 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.551402    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 31 01:20:45.561510 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561491    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cbf07f-8927-4379-9283-409f889cdf1e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561603 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561518    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-credential-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561603 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561550    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561603 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561569    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cbf07f-8927-4379-9283-409f889cdf1e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561725 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561660    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-envoy\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561725 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561681    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-certs\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561725 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561696    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-data\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561725 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561711    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-istio-token\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.561911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.561726    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5qv4\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-kube-api-access-k5qv4\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.567435 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.567415    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-bdg2g\""
Oct 31 01:20:45.569182 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.569159    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"]
Oct 31 01:20:45.662610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662572    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.662610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662621    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cbf07f-8927-4379-9283-409f889cdf1e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.662863 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662657    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-envoy\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.662863 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662687    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-certs\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.662863 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662708    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-data\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663027 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662859    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-istio-token\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663027 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662895    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5qv4\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-kube-api-access-k5qv4\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663027 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.662996    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cbf07f-8927-4379-9283-409f889cdf1e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663027 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.663016    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663027 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.663022    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-credential-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663506 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.663476    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-certs\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663607 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.663507    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-data\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.663790 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.663771    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-credential-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.664060 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.664038    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cbf07f-8927-4379-9283-409f889cdf1e-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.665164 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.665148    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-envoy\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.665614 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.665593    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cbf07f-8927-4379-9283-409f889cdf1e-istio-podinfo\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.723278 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.723220    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5qv4\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-kube-api-access-k5qv4\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.723278 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.723243    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-istio-token\") pod \"router-gateway-2-openshift-default-7fccdfc75c-h52q5\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:45.782452 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:45.782431    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:46.207939 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:46.207907    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"]
Oct 31 01:20:46.215763 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:20:46.215734    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cbf07f_8927_4379_9283_409f889cdf1e.slice/crio-24d3820c14dce18ed65bb63596e9355988a6a730e6c8866685e88204aff016ec WatchSource:0}: Error finding container 24d3820c14dce18ed65bb63596e9355988a6a730e6c8866685e88204aff016ec: Status 404 returned error can't find the container with id 24d3820c14dce18ed65bb63596e9355988a6a730e6c8866685e88204aff016ec
Oct 31 01:20:46.217719 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:46.217688    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:20:46.217788 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:46.217758    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:20:46.217821 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:46.217790    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:20:47.100544 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:47.100510    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" event={"ID":"98cbf07f-8927-4379-9283-409f889cdf1e","Type":"ContainerStarted","Data":"8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa"}
Oct 31 01:20:47.100914 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:47.100562    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" event={"ID":"98cbf07f-8927-4379-9283-409f889cdf1e","Type":"ContainerStarted","Data":"24d3820c14dce18ed65bb63596e9355988a6a730e6c8866685e88204aff016ec"}
Oct 31 01:20:47.263638 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:47.263588    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" podStartSLOduration=2.26357582 podStartE2EDuration="2.26357582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:20:47.263195519 +0000 UTC m=+3055.818082955" watchObservedRunningTime="2026-10-31 01:20:47.26357582 +0000 UTC m=+3055.818463259"
Oct 31 01:20:47.783584 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:47.783545    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:47.788460 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:47.788437    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:48.106259 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:48.106171    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:48.107141 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:48.107117    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:20:48.552546 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:20:48.552514    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:20:56.064727 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.064692    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"]
Oct 31 01:20:56.069506 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.069482    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.083031 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.083002    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"]
Oct 31 01:20:56.128284 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.128224    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 31 01:20:56.128284 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.128225    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:20:56.128284 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.128233    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-7hqmk\""
Oct 31 01:20:56.128935 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.128914    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 31 01:20:56.170622 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.170590    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.170622 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.170624    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e547ed-75ff-4114-a3ec-37fcd852d802-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.170868 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.170652    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e547ed-75ff-4114-a3ec-37fcd852d802-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.170868 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.170779    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.170868 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.170809    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcgrg\" (UniqueName: \"kubernetes.io/projected/f0e547ed-75ff-4114-a3ec-37fcd852d802-kube-api-access-qcgrg\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.170868 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.170848    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.171057 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.170960    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-home\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.171103 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.171077    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272231 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272197    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e547ed-75ff-4114-a3ec-37fcd852d802-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272455 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272243    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272455 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272259    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcgrg\" (UniqueName: \"kubernetes.io/projected/f0e547ed-75ff-4114-a3ec-37fcd852d802-kube-api-access-qcgrg\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272455 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272279    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272455 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272296    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-home\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272687 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272504    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272687 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272579    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272687 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272611    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e547ed-75ff-4114-a3ec-37fcd852d802-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272846 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272724    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272846 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272818    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-home\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.272986 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272961    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.273049 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.272983    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.273487 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.273462    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e547ed-75ff-4114-a3ec-37fcd852d802-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.274809 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.274778    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.274987 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.274967    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e547ed-75ff-4114-a3ec-37fcd852d802-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.332855 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.332766    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcgrg\" (UniqueName: \"kubernetes.io/projected/f0e547ed-75ff-4114-a3ec-37fcd852d802-kube-api-access-qcgrg\") pod \"router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.386520 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.386482    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:20:56.829020 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:56.828971    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"]
Oct 31 01:20:56.835396 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:20:56.835333    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e547ed_75ff_4114_a3ec_37fcd852d802.slice/crio-348de47423157aa2d0b440cb2e609e5f4300d3586672d0db3cb47ef98b72cb7a WatchSource:0}: Error finding container 348de47423157aa2d0b440cb2e609e5f4300d3586672d0db3cb47ef98b72cb7a: Status 404 returned error can't find the container with id 348de47423157aa2d0b440cb2e609e5f4300d3586672d0db3cb47ef98b72cb7a
Oct 31 01:20:57.167605 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:57.167530    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" event={"ID":"f0e547ed-75ff-4114-a3ec-37fcd852d802","Type":"ContainerStarted","Data":"2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8"}
Oct 31 01:20:57.167605 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:57.167580    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" event={"ID":"f0e547ed-75ff-4114-a3ec-37fcd852d802","Type":"ContainerStarted","Data":"348de47423157aa2d0b440cb2e609e5f4300d3586672d0db3cb47ef98b72cb7a"}
Oct 31 01:20:59.181187 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.181153    4888 generic.go:356] "Generic (PLEG): container finished" podID="98cbf07f-8927-4379-9283-409f889cdf1e" containerID="8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa" exitCode=0
Oct 31 01:20:59.181664 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.181221    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" event={"ID":"98cbf07f-8927-4379-9283-409f889cdf1e","Type":"ContainerDied","Data":"8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa"}
Oct 31 01:20:59.181664 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.181602    4888 scope.go:122] "RemoveContainer" containerID="8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa"
Oct 31 01:20:59.182710 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.182682    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:20:59.182806 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.182755    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:20:59.182806 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.182788    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:20:59.182940 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.182923    4888 generic.go:356] "Generic (PLEG): container finished" podID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerID="2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8" exitCode=0
Oct 31 01:20:59.182980 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:20:59.182958    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" event={"ID":"f0e547ed-75ff-4114-a3ec-37fcd852d802","Type":"ContainerDied","Data":"2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8"}
Oct 31 01:21:00.189197 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:00.189151    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" event={"ID":"f0e547ed-75ff-4114-a3ec-37fcd852d802","Type":"ContainerStarted","Data":"6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812"}
Oct 31 01:21:00.191006 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:00.190977    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" event={"ID":"98cbf07f-8927-4379-9283-409f889cdf1e","Type":"ContainerStarted","Data":"91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b"}
Oct 31 01:21:00.308922 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:00.308862    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podStartSLOduration=3.17140037 podStartE2EDuration="5.308847802s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.137447432s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:00.307205716 +0000 UTC m=+3068.862093187" watchObservedRunningTime="2026-10-31 01:21:00.308847802 +0000 UTC m=+3068.863735240"
Oct 31 01:21:00.783456 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:00.783417    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:21:00.784676 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:00.784653    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:21:01.783117 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:01.783072    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:21:02.783281 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:02.783234    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:21:03.106509 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:03.106418    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:21:03.783600 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:03.783552    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:21:04.783720 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:04.783673    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 31 01:21:05.786912 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:05.786875    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:21:05.787786 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:05.787765    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:21:06.387613 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:06.387533    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:21:06.387613 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:06.387604    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:21:06.389020 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:06.388984    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:21:16.387171 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:16.387131    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:21:18.552187 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:21:18.552151    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:21:26.387558 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:26.387461    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:21:36.388026 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:36.387984    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:21:46.387594 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:46.387556    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:21:48.551851 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:21:48.551808    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:21:49.875721 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.875687    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"]
Oct 31 01:21:49.878952 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.878932    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967362 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967314    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967539 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967382    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967539 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967514    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967660 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967555    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967660 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967592    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdzxr\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-kube-api-access-rdzxr\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967744 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967649    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967744 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967687    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967744 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967713    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:49.967892 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:49.967796    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069101 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069055    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069133    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069182    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069215    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069253    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdzxr\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-kube-api-access-rdzxr\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069287    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069314    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069341    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069413    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069770 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069645    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.069770 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.069675    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.070023 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.070001    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.070104 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.070033    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.070503 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.070477    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.071925 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.071902    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.072086 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.072068    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.087628 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.087599    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"]
Oct 31 01:21:50.129650 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.129566    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.129650 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.129575    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdzxr\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-kube-api-access-rdzxr\") pod \"openshift-ai-inference-openshift-default-568c495856-77jcv\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.191586 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.191548    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:50.620106 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.620066    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"]
Oct 31 01:21:50.628166 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:21:50.628135    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3651906b_d2d2_4e06_a8ce_8a6b0349c55b.slice/crio-144969724a99343e197caa34f0d5e4ace1b8af0b3af293fa472fc15d64997c3e WatchSource:0}: Error finding container 144969724a99343e197caa34f0d5e4ace1b8af0b3af293fa472fc15d64997c3e: Status 404 returned error can't find the container with id 144969724a99343e197caa34f0d5e4ace1b8af0b3af293fa472fc15d64997c3e
Oct 31 01:21:50.630200 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.630169    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:21:50.630279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.630239    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:21:50.630279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:50.630266    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:21:51.446112 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:51.446073    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" event={"ID":"3651906b-d2d2-4e06-a8ce-8a6b0349c55b","Type":"ContainerStarted","Data":"6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358"}
Oct 31 01:21:51.446535 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:51.446125    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" event={"ID":"3651906b-d2d2-4e06-a8ce-8a6b0349c55b","Type":"ContainerStarted","Data":"144969724a99343e197caa34f0d5e4ace1b8af0b3af293fa472fc15d64997c3e"}
Oct 31 01:21:51.615374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:51.612372    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podStartSLOduration=2.612336668 podStartE2EDuration="2.612336668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:21:51.609554482 +0000 UTC m=+3120.164441949" watchObservedRunningTime="2026-10-31 01:21:51.612336668 +0000 UTC m=+3120.167224106"
Oct 31 01:21:52.192752 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:52.192715    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:52.194152 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:52.194126    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-77jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:21:52.194280 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:52.194184    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 01:21:53.193014 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:53.192978    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-77jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:21:53.193458 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:53.193061    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 01:21:54.192733 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:54.192690    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-77jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:21:54.192945 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:54.192753    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 01:21:55.192184 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:55.192134    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-77jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:21:55.192708 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:55.192204    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 01:21:56.192823 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:56.192788    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-77jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 31 01:21:56.193273 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:56.192854    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 31 01:21:56.386995 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:56.386957    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:21:57.196457 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:57.196427    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:57.196905 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:57.196689    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:21:57.197458 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:21:57.197436    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:22:05.001362 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.001304    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"]
Oct 31 01:22:05.006009 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.005976    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.112960 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.112923    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.112967    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.112994    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113128 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.113110    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113236 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.113141    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113236 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.113159    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sss99\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-kube-api-access-sss99\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113297 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.113259    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113297 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.113282    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.113387 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.113300    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.206678 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.206642    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"]
Oct 31 01:22:05.214222 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214195    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214392 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214247    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214392 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214274    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214392 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214337    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214392 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214386    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214629 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214414    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sss99\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-kube-api-access-sss99\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214629 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214468    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214629 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214504    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214629 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214531    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214817 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214647    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214817 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214801    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.214946 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.214933    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.215150 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.215124    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.215663 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.215639    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.216974 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.216948    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.217502 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.217479    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.275074 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.274995    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.275227 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.275158    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sss99\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-kube-api-access-sss99\") pod \"openshift-ai-inference-openshift-default-568c495856-jtzw2\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.318769 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.318745    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:05.754911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.754872    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"]
Oct 31 01:22:05.762182 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:22:05.762156    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1cbb4b_7cc6_4dd1_908f_6c3d353ce8b7.slice/crio-fd7905e77499c97e985970375490b843b293562eca72cd227a3465177db61a87 WatchSource:0}: Error finding container fd7905e77499c97e985970375490b843b293562eca72cd227a3465177db61a87: Status 404 returned error can't find the container with id fd7905e77499c97e985970375490b843b293562eca72cd227a3465177db61a87
Oct 31 01:22:05.764404 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.764370    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:22:05.764479 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.764441    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:22:05.764479 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:05.764473    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:22:06.386916 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:06.386878    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:22:06.543924 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:06.543883    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" event={"ID":"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7","Type":"ContainerStarted","Data":"8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9"}
Oct 31 01:22:06.544103 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:06.543958    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" event={"ID":"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7","Type":"ContainerStarted","Data":"fd7905e77499c97e985970375490b843b293562eca72cd227a3465177db61a87"}
Oct 31 01:22:06.701017 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:06.700964    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podStartSLOduration=2.70095132 podStartE2EDuration="2.70095132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:06.698757724 +0000 UTC m=+3135.253645175" watchObservedRunningTime="2026-10-31 01:22:06.70095132 +0000 UTC m=+3135.255838758"
Oct 31 01:22:07.321985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:07.321951    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:07.322456 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:07.322433    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-jtzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:22:07.322619 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:07.322594    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 01:22:08.319796 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:08.319755    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-jtzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:22:08.320421 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:08.320385    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 01:22:09.322768 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:09.322730    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-jtzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:22:09.323438 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:09.323395    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 01:22:10.320376 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:10.319782    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-jtzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:22:10.320376 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:10.319849    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 01:22:11.319789 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:11.319750    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-jtzw2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 31 01:22:11.320430 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:11.320399    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 31 01:22:12.323596 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:12.323562    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:12.324054 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:12.324034    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:12.324787 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:12.324767    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:22:16.387789 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:16.387739    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:22:18.551699 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:22:18.551663    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:22:20.123498 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.123460    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"]
Oct 31 01:22:20.126930 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.126905    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.152911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.152890    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153029 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.152922    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153029 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.152956    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153146 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.153112    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153207 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.153166    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153207 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.153192    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153594 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.153550    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153733 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.153681    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56zzm\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-kube-api-access-56zzm\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.153808 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.153783    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.228435 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.228395    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"]
Oct 31 01:22:20.254675 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254646    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.254843 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254700    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.254843 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254726    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.254843 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254793    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.254843 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254831    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56zzm\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-kube-api-access-56zzm\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255065 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254886    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255065 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254913    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255065 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254950    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255065 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.254990    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255272 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.255091    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255407 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.255382    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255545 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.255522    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255779 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.255755    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.255942 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.255918    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.257312 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.257292    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.257571 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.257551    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.314743 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.314707    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.314933 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.314838    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56zzm\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-kube-api-access-56zzm\") pod \"openshift-ai-inference-openshift-default-568c495856-zckrp\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.440258 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.440167    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:20.870082 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.870039    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"]
Oct 31 01:22:20.876268 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:22:20.876239    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54c71c62_a915_4baa_b60f_a8ff61c56dbf.slice/crio-c186e186384e36d1e12ad25042cebf79168a9f60671dc37c15f9e895b4c2188f WatchSource:0}: Error finding container c186e186384e36d1e12ad25042cebf79168a9f60671dc37c15f9e895b4c2188f: Status 404 returned error can't find the container with id c186e186384e36d1e12ad25042cebf79168a9f60671dc37c15f9e895b4c2188f
Oct 31 01:22:20.878290 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.878254    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:22:20.878399 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.878318    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:22:20.878399 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:20.878362    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:22:21.645443 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:21.644863    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" event={"ID":"54c71c62-a915-4baa-b60f-a8ff61c56dbf","Type":"ContainerStarted","Data":"5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4"}
Oct 31 01:22:21.645443 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:21.644928    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" event={"ID":"54c71c62-a915-4baa-b60f-a8ff61c56dbf","Type":"ContainerStarted","Data":"c186e186384e36d1e12ad25042cebf79168a9f60671dc37c15f9e895b4c2188f"}
Oct 31 01:22:21.815376 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:21.812798    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podStartSLOduration=1.812776468 podStartE2EDuration="1.812776468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:22:21.810247855 +0000 UTC m=+3150.365135293" watchObservedRunningTime="2026-10-31 01:22:21.812776468 +0000 UTC m=+3150.367663907"
Oct 31 01:22:22.441188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:22.441146    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:22.442888 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:22.442856    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-zckrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:22:22.443047 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:22.442916    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:22:23.440808 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:23.440763    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-zckrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:22:23.441374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:23.440830    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:22:24.440928 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:24.440874    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-zckrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:22:24.441463 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:24.440952    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:22:25.441080 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:25.441047    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-zckrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:22:25.441629 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:25.441110    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:22:26.387630 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:26.387581    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:22:26.441143 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:26.441097    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-zckrp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 31 01:22:26.441716 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:26.441164    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 31 01:22:27.444608 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:27.444575    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:27.445113 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:27.444952    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:27.445682 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:27.445667    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:22:36.387845 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:36.387794    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:22:46.386854 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:46.386800    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": dial tcp 10.134.0.69:8000: connect: connection refused"
Oct 31 01:22:48.552074 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:22:48.552038    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:22:56.397304 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:56.397218    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:22:56.404823 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:22:56.404796    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:23:12.785269 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:12.785234    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"]
Oct 31 01:23:18.551991 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:23:18.551958    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:23:27.808175 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:27.808129    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main" containerID="cri-o://6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812" gracePeriod=45
Oct 31 01:23:28.486394 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.486368    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:23:28.559753 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.559668    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-tmp-dir\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.559753 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.559715    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0e547ed-75ff-4114-a3ec-37fcd852d802-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e547ed-75ff-4114-a3ec-37fcd852d802-cabundle-cert\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.559753 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.559750    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e547ed-75ff-4114-a3ec-37fcd852d802-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e547ed-75ff-4114-a3ec-37fcd852d802-tls-certs\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.560033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.559767    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e547ed-75ff-4114-a3ec-37fcd852d802-kube-api-access-qcgrg\" (UniqueName: \"kubernetes.io/projected/f0e547ed-75ff-4114-a3ec-37fcd852d802-kube-api-access-qcgrg\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.560033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.559798    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-home\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-home\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.560033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.559851    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-kserve-provision-location\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.560033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.559975    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-dshm\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.560033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.560013    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-model-cache\") pod \"f0e547ed-75ff-4114-a3ec-37fcd852d802\" (UID: \"f0e547ed-75ff-4114-a3ec-37fcd852d802\") "
Oct 31 01:23:28.560304 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.560176    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e547ed-75ff-4114-a3ec-37fcd852d802-cabundle-cert" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:23:28.560391 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.560338    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0e547ed-75ff-4114-a3ec-37fcd852d802-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.560537 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.560484    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-model-cache" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.560537 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.560528    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-home" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.562311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.562281    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e547ed-75ff-4114-a3ec-37fcd852d802-tls-certs" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:23:28.562815 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.562788    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e547ed-75ff-4114-a3ec-37fcd852d802-kube-api-access-qcgrg" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "kube-api-access-qcgrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:23:28.562815 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.562795    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-dshm" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.571878 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.571854    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-tmp-dir" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.589494 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.589470    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-kserve-provision-location" pod "f0e547ed-75ff-4114-a3ec-37fcd852d802" (UID: "f0e547ed-75ff-4114-a3ec-37fcd852d802"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:23:28.661405 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.661373    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.661405 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.661404    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.661620 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.661419    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.661620 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.661432    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.661620 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.661446    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0e547ed-75ff-4114-a3ec-37fcd852d802-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.661620 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.661459    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0e547ed-75ff-4114-a3ec-37fcd852d802-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.661620 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.661471    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcgrg\" (UniqueName: \"kubernetes.io/projected/f0e547ed-75ff-4114-a3ec-37fcd852d802-kube-api-access-qcgrg\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:23:28.979111 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.979067    4888 generic.go:356] "Generic (PLEG): container finished" podID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerID="6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812" exitCode=0
Oct 31 01:23:28.979553 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.979127    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"
Oct 31 01:23:28.979553 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.979149    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" event={"ID":"f0e547ed-75ff-4114-a3ec-37fcd852d802","Type":"ContainerDied","Data":"6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812"}
Oct 31 01:23:28.979553 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.979208    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6" event={"ID":"f0e547ed-75ff-4114-a3ec-37fcd852d802","Type":"ContainerDied","Data":"348de47423157aa2d0b440cb2e609e5f4300d3586672d0db3cb47ef98b72cb7a"}
Oct 31 01:23:28.979553 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.979229    4888 scope.go:122] "RemoveContainer" containerID="6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812"
Oct 31 01:23:28.991586 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.991553    4888 scope.go:122] "RemoveContainer" containerID="2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8"
Oct 31 01:23:28.998891 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.998875    4888 scope.go:122] "RemoveContainer" containerID="6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812"
Oct 31 01:23:28.999132 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:23:28.999113    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812\": container with ID starting with 6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812 not found: ID does not exist" containerID="6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812"
Oct 31 01:23:28.999176 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.999142    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812"} err="failed to get container status \"6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812\": rpc error: code = NotFound desc = could not find container \"6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812\": container with ID starting with 6f9c64d318c243efc6ca4cecc4cce0a7a16cb99929b504f3afa1b091f3dbf812 not found: ID does not exist"
Oct 31 01:23:28.999176 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.999156    4888 scope.go:122] "RemoveContainer" containerID="2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8"
Oct 31 01:23:28.999423 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:23:28.999408    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8\": container with ID starting with 2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8 not found: ID does not exist" containerID="2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8"
Oct 31 01:23:28.999489 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:28.999429    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8"} err="failed to get container status \"2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8\": rpc error: code = NotFound desc = could not find container \"2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8\": container with ID starting with 2ef619155f23597d55ac195ed3415c1d9cd5940fd91f1bd561c24d54e94321a8 not found: ID does not exist"
Oct 31 01:23:29.139808 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:29.139777    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"]
Oct 31 01:23:29.146685 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:29.146658    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-prefill-7867f4f5f7-hfbz6"]
Oct 31 01:23:29.553539 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:23:29.553510    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" path="/var/lib/kubelet/pods/f0e547ed-75ff-4114-a3ec-37fcd852d802/volumes"
Oct 31 01:23:48.551971 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:23:48.551930    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:24:07.210175 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:07.210138    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"]
Oct 31 01:24:07.210780 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:07.210363    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy" containerID="cri-o://91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b" gracePeriod=30
Oct 31 01:24:12.459575 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.459550    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:24:12.505944 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.505874    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-kube-api-access-k5qv4\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-kube-api-access-k5qv4\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.505944 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.505924    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-credential-socket\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506116 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506001    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98cbf07f-8927-4379-9283-409f889cdf1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cbf07f-8927-4379-9283-409f889cdf1e-istiod-ca-cert\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506116 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506029    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-envoy\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506116 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506053    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-data\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506276 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506112    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-certs\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506276 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506145    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98cbf07f-8927-4379-9283-409f889cdf1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cbf07f-8927-4379-9283-409f889cdf1e-istio-podinfo\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506276 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506172    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-socket\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506276 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506200    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-istio-token\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-istio-token\") pod \"98cbf07f-8927-4379-9283-409f889cdf1e\" (UID: \"98cbf07f-8927-4379-9283-409f889cdf1e\") "
Oct 31 01:24:12.506276 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506221    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-credential-socket" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:12.506543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506465    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-certs" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:12.506543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506470    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-data" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:12.506543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506482    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.506692 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506550    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cbf07f-8927-4379-9283-409f889cdf1e-istiod-ca-cert" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:24:12.506692 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.506681    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-socket" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:12.508570 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.508548    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98cbf07f-8927-4379-9283-409f889cdf1e-istio-podinfo" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:24:12.508718 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.508610    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-kube-api-access-k5qv4" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "kube-api-access-k5qv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:12.508718 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.508685    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-istio-token" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:24:12.508972 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.508953    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-envoy" pod "98cbf07f-8927-4379-9283-409f889cdf1e" (UID: "98cbf07f-8927-4379-9283-409f889cdf1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:24:12.607490 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607461    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.607490 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607489    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98cbf07f-8927-4379-9283-409f889cdf1e-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.607706 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607502    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.607706 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607511    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.607706 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607519    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5qv4\" (UniqueName: \"kubernetes.io/projected/98cbf07f-8927-4379-9283-409f889cdf1e-kube-api-access-k5qv4\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.607706 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607529    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98cbf07f-8927-4379-9283-409f889cdf1e-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.607706 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607537    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:12.607706 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:12.607545    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98cbf07f-8927-4379-9283-409f889cdf1e-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:24:13.183200 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.183165    4888 generic.go:356] "Generic (PLEG): container finished" podID="98cbf07f-8927-4379-9283-409f889cdf1e" containerID="91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b" exitCode=0
Oct 31 01:24:13.183423 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.183225    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"
Oct 31 01:24:13.183423 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.183256    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" event={"ID":"98cbf07f-8927-4379-9283-409f889cdf1e","Type":"ContainerDied","Data":"91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b"}
Oct 31 01:24:13.183423 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.183325    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5" event={"ID":"98cbf07f-8927-4379-9283-409f889cdf1e","Type":"ContainerDied","Data":"24d3820c14dce18ed65bb63596e9355988a6a730e6c8866685e88204aff016ec"}
Oct 31 01:24:13.183423 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.183384    4888 scope.go:122] "RemoveContainer" containerID="91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b"
Oct 31 01:24:13.192649 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.192633    4888 scope.go:122] "RemoveContainer" containerID="8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa"
Oct 31 01:24:13.200446 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.200431    4888 scope.go:122] "RemoveContainer" containerID="91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b"
Oct 31 01:24:13.200714 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:24:13.200694    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b\": container with ID starting with 91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b not found: ID does not exist" containerID="91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b"
Oct 31 01:24:13.200764 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.200722    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b"} err="failed to get container status \"91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b\": rpc error: code = NotFound desc = could not find container \"91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b\": container with ID starting with 91ad39f5d4588b408dffe6ce7d459cab775f956f619fe7c744ef14aa576a065b not found: ID does not exist"
Oct 31 01:24:13.200764 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.200740    4888 scope.go:122] "RemoveContainer" containerID="8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa"
Oct 31 01:24:13.200956 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:24:13.200937    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa\": container with ID starting with 8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa not found: ID does not exist" containerID="8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa"
Oct 31 01:24:13.200996 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.200960    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa"} err="failed to get container status \"8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa\": rpc error: code = NotFound desc = could not find container \"8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa\": container with ID starting with 8c2c4dc930cae0a307f5d2839091944177c1523b5ad573a26c850ee9e4ad1caa not found: ID does not exist"
Oct 31 01:24:13.391875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.391838    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"]
Oct 31 01:24:13.438788 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.438709    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-h52q5"]
Oct 31 01:24:13.553875 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:13.553840    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" path="/var/lib/kubelet/pods/98cbf07f-8927-4379-9283-409f889cdf1e/volumes"
Oct 31 01:24:18.553805 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:24:18.553769    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:24:48.556852 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:24:48.556809    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:24:51.901995 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:51.901965    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:24:51.916176 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:24:51.916157    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:25:18.551864 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:25:18.551826    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:25:48.551237 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:25:48.551202    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:26:18.551921 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:26:18.551884    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:26:40.298546 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298510    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"]
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298829    4888 cpu_manager.go:414] "containerMap: removing container" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="storage-initializer"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298840    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="storage-initializer"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298856    4888 cpu_manager.go:414] "containerMap: removing container" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298861    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298879    4888 cpu_manager.go:414] "containerMap: removing container" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298885    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298892    4888 cpu_manager.go:414] "containerMap: removing container" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298897    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298959    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e547ed-75ff-4114-a3ec-37fcd852d802" containerName="main"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298971    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy"
Oct 31 01:26:40.299077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.298978    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="98cbf07f-8927-4379-9283-409f889cdf1e" containerName="istio-proxy"
Oct 31 01:26:40.301895 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.301876    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.355921 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.355884    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-rspps\""
Oct 31 01:26:40.357519 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.357494    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-s5g94\""
Oct 31 01:26:40.402831 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.402780    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 31 01:26:40.402831 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.402816    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 31 01:26:40.404527 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.404509    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"]
Oct 31 01:26:40.422130 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.422103    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe220633-7e8e-4e64-9774-cd12deab16f5-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.422268 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.422221    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gts\" (UniqueName: \"kubernetes.io/projected/fe220633-7e8e-4e64-9774-cd12deab16f5-kube-api-access-69gts\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.456041 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.456014    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 31 01:26:40.523442 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.523403    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe220633-7e8e-4e64-9774-cd12deab16f5-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.523625 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.523503    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gts\" (UniqueName: \"kubernetes.io/projected/fe220633-7e8e-4e64-9774-cd12deab16f5-kube-api-access-69gts\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.526073 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.526048    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe220633-7e8e-4e64-9774-cd12deab16f5-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.551000 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.550940    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 01:26:40.554531 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.554509    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.583800 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.583775    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gts\" (UniqueName: \"kubernetes.io/projected/fe220633-7e8e-4e64-9774-cd12deab16f5-kube-api-access-69gts\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.605795 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.605770    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:26:40.605927 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.605796    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 31 01:26:40.605927 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.605823    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-zvgr4\""
Oct 31 01:26:40.614091 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.614071    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:40.614477 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.614457    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 01:26:40.624429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624405    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.624503 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624462    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.624608 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624584    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.624645 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624623    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwdz7\" (UniqueName: \"kubernetes.io/projected/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kube-api-access-zwdz7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.624720 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624689    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.624845 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624810    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.624912 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624860    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.624912 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.624890    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725295 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725262    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725493 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725327    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725493 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725396    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725493 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725439    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725493 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725463    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwdz7\" (UniqueName: \"kubernetes.io/projected/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kube-api-access-zwdz7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725504    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725550    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725578    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725761    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.725879 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725784    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.726152 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.725990    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.726152 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.726116    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.726436 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.726226    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.727916 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.727889    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.728010 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.727997    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.785341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.785312    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwdz7\" (UniqueName: \"kubernetes.io/projected/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kube-api-access-zwdz7\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.869695 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.869623    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:26:40.989187 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:40.989152    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"]
Oct 31 01:26:40.996277 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:26:40.996249    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe220633_7e8e_4e64_9774_cd12deab16f5.slice/crio-c1dd6f5257d8eff124c8a34b601fddf40ec1086ae54047ae718827ffde1e6b03 WatchSource:0}: Error finding container c1dd6f5257d8eff124c8a34b601fddf40ec1086ae54047ae718827ffde1e6b03: Status 404 returned error can't find the container with id c1dd6f5257d8eff124c8a34b601fddf40ec1086ae54047ae718827ffde1e6b03
Oct 31 01:26:41.244771 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:41.244736    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 01:26:41.251932 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:26:41.251906    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc9247f_ffe9_412e_ab71_65e0d3fd924c.slice/crio-ae2dea799bf12c329d6a9b4eb0d499d4e3871824bebb5ebc7f06407f68d51d03 WatchSource:0}: Error finding container ae2dea799bf12c329d6a9b4eb0d499d4e3871824bebb5ebc7f06407f68d51d03: Status 404 returned error can't find the container with id ae2dea799bf12c329d6a9b4eb0d499d4e3871824bebb5ebc7f06407f68d51d03
Oct 31 01:26:41.878563 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:41.878523    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" event={"ID":"fe220633-7e8e-4e64-9774-cd12deab16f5","Type":"ContainerStarted","Data":"bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d"}
Oct 31 01:26:41.879042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:41.878593    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:26:41.879042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:41.878618    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" event={"ID":"fe220633-7e8e-4e64-9774-cd12deab16f5","Type":"ContainerStarted","Data":"c1dd6f5257d8eff124c8a34b601fddf40ec1086ae54047ae718827ffde1e6b03"}
Oct 31 01:26:41.880238 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:41.880209    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"0cc9247f-ffe9-412e-ab71-65e0d3fd924c","Type":"ContainerStarted","Data":"4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625"}
Oct 31 01:26:41.880368 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:41.880256    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"0cc9247f-ffe9-412e-ab71-65e0d3fd924c","Type":"ContainerStarted","Data":"ae2dea799bf12c329d6a9b4eb0d499d4e3871824bebb5ebc7f06407f68d51d03"}
Oct 31 01:26:41.992224 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:41.992172    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podStartSLOduration=1.992159897 podStartE2EDuration="1.992159897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:41.991190295 +0000 UTC m=+3410.546077732" watchObservedRunningTime="2026-10-31 01:26:41.992159897 +0000 UTC m=+3410.547047334"
Oct 31 01:26:43.892212 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:43.892126    4888 generic.go:356] "Generic (PLEG): container finished" podID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerID="4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625" exitCode=0
Oct 31 01:26:43.892608 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:43.892267    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"0cc9247f-ffe9-412e-ab71-65e0d3fd924c","Type":"ContainerDied","Data":"4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625"}
Oct 31 01:26:44.899030 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:44.898979    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"0cc9247f-ffe9-412e-ab71-65e0d3fd924c","Type":"ContainerStarted","Data":"a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f"}
Oct 31 01:26:45.019009 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:45.018948    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.840135192 podStartE2EDuration="5.018933571s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.178798379s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:26:45.018558481 +0000 UTC m=+3413.573445918" watchObservedRunningTime="2026-10-31 01:26:45.018933571 +0000 UTC m=+3413.573821001"
Oct 31 01:26:48.552459 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:26:48.552422    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:26:52.163766 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:52.163731    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"]
Oct 31 01:26:52.164174 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:52.164005    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy" containerID="cri-o://6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358" gracePeriod=30
Oct 31 01:26:52.210551 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:52.210512    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"]
Oct 31 01:26:52.210744 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:52.210722    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy" containerID="cri-o://5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4" gracePeriod=30
Oct 31 01:26:57.199604 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.199569    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-77jcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": read tcp 10.134.0.2:35302->10.134.0.70:15021: read: connection reset by peer" start-of-body=
Oct 31 01:26:57.200063 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.199631    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": read tcp 10.134.0.2:35302->10.134.0.70:15021: read: connection reset by peer"
Oct 31 01:26:57.453372 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.453298    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:26:57.488938 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.488915    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:26:57.590173 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590140    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-kube-api-access-56zzm\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-kube-api-access-56zzm\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590184    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-credential-socket\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590210    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-socket\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590233    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-data\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590258    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-envoy\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.590378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590289    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-certs\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590313    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-data\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.590378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590343    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-podinfo\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590419    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-token\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-token\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590449    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istiod-ca-cert\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590487    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-kube-api-access-rdzxr\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-kube-api-access-rdzxr\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590537    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-token\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-token\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590566    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-certs\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590604    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istiod-ca-cert\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590596    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-data" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590632    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-envoy\") pod \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\" (UID: \"54c71c62-a915-4baa-b60f-a8ff61c56dbf\") "
Oct 31 01:26:57.590735 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590728    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-podinfo\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.591170 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590755    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-socket\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.591170 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.590786    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-credential-socket\") pod \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\" (UID: \"3651906b-d2d2-4e06-a8ce-8a6b0349c55b\") "
Oct 31 01:26:57.591170 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.591048    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.591170 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.591087    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-credential-socket" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.591384 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.591252    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-credential-socket" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.591384 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.591321    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-data" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.591501 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.591462    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istiod-ca-cert" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:57.592433 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.591639    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-certs" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.592433 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.592016    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istiod-ca-cert" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:26:57.592618 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.592455    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-socket" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.592713 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.592644    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-socket" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.592911 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.592884    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-kube-api-access-56zzm" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "kube-api-access-56zzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:57.593030 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.593003    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-envoy" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.593135 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.593027    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-kube-api-access-rdzxr" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "kube-api-access-rdzxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:57.593374 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.593337    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-certs" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.593851 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.593830    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-token" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:57.594048 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.594035    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-envoy" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:26:57.594141 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.594128    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-token" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:26:57.595105 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.595084    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-podinfo" pod "3651906b-d2d2-4e06-a8ce-8a6b0349c55b" (UID: "3651906b-d2d2-4e06-a8ce-8a6b0349c55b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:57.595301 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.595288    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-podinfo" pod "54c71c62-a915-4baa-b60f-a8ff61c56dbf" (UID: "54c71c62-a915-4baa-b60f-a8ff61c56dbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:26:57.691495 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691460    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691495 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691488    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691495 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691502    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdzxr\" (UniqueName: \"kubernetes.io/projected/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-kube-api-access-rdzxr\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691516    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691527    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691539    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691550    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691561    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691575    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691587    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691599    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56zzm\" (UniqueName: \"kubernetes.io/projected/54c71c62-a915-4baa-b60f-a8ff61c56dbf-kube-api-access-56zzm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691611    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691622    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691637    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691649    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3651906b-d2d2-4e06-a8ce-8a6b0349c55b-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691661    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54c71c62-a915-4baa-b60f-a8ff61c56dbf-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.691751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.691673    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54c71c62-a915-4baa-b60f-a8ff61c56dbf-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:26:57.967487 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.967458    4888 generic.go:356] "Generic (PLEG): container finished" podID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerID="6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358" exitCode=0
Oct 31 01:26:57.967662 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.967509    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"
Oct 31 01:26:57.967662 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.967549    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" event={"ID":"3651906b-d2d2-4e06-a8ce-8a6b0349c55b","Type":"ContainerDied","Data":"6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358"}
Oct 31 01:26:57.967662 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.967602    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv" event={"ID":"3651906b-d2d2-4e06-a8ce-8a6b0349c55b","Type":"ContainerDied","Data":"144969724a99343e197caa34f0d5e4ace1b8af0b3af293fa472fc15d64997c3e"}
Oct 31 01:26:57.967662 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.967628    4888 scope.go:122] "RemoveContainer" containerID="6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358"
Oct 31 01:26:57.969104 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.969076    4888 generic.go:356] "Generic (PLEG): container finished" podID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerID="5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4" exitCode=0
Oct 31 01:26:57.969216 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.969106    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" event={"ID":"54c71c62-a915-4baa-b60f-a8ff61c56dbf","Type":"ContainerDied","Data":"5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4"}
Oct 31 01:26:57.969216 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.969139    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"
Oct 31 01:26:57.969216 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.969168    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" event={"ID":"54c71c62-a915-4baa-b60f-a8ff61c56dbf","Type":"ContainerDied","Data":"c186e186384e36d1e12ad25042cebf79168a9f60671dc37c15f9e895b4c2188f"}
Oct 31 01:26:57.977221 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.977202    4888 scope.go:122] "RemoveContainer" containerID="6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358"
Oct 31 01:26:57.977528 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:26:57.977509    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358\": container with ID starting with 6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358 not found: ID does not exist" containerID="6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358"
Oct 31 01:26:57.977632 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.977540    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358"} err="failed to get container status \"6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358\": rpc error: code = NotFound desc = could not find container \"6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358\": container with ID starting with 6f35b2e9dd008f415240ee2792a0607c649d16bd3622be6892682daf5b4d4358 not found: ID does not exist"
Oct 31 01:26:57.977632 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.977562    4888 scope.go:122] "RemoveContainer" containerID="5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4"
Oct 31 01:26:57.985513 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.985495    4888 scope.go:122] "RemoveContainer" containerID="5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4"
Oct 31 01:26:57.985796 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:26:57.985772    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4\": container with ID starting with 5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4 not found: ID does not exist" containerID="5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4"
Oct 31 01:26:57.985864 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:57.985808    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4"} err="failed to get container status \"5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4\": rpc error: code = NotFound desc = could not find container \"5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4\": container with ID starting with 5421ee52b4c7e03f3821f945ec85e74ea11d5e5b49766a31a316dbe0af81f3e4 not found: ID does not exist"
Oct 31 01:26:58.126764 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:58.126731    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"]
Oct 31 01:26:58.173260 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:58.173222    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp"]
Oct 31 01:26:58.238462 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:58.238378    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"]
Oct 31 01:26:58.244834 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:58.244810    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-77jcv"]
Oct 31 01:26:58.446437 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:58.446407    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-zckrp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 01:26:58.446606 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:58.446467    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-zckrp" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:26:59.553539 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:59.553506    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" path="/var/lib/kubelet/pods/3651906b-d2d2-4e06-a8ce-8a6b0349c55b/volumes"
Oct 31 01:26:59.553923 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:26:59.553907    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" path="/var/lib/kubelet/pods/54c71c62-a915-4baa-b60f-a8ff61c56dbf/volumes"
Oct 31 01:27:12.886583 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:27:12.886555    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:27:18.552926 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:27:18.552883    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:27:48.552015 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:27:48.551979    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:28:18.551763 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:28:18.551726    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:28:20.616693 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:20.616657    4888 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:28:20.650209 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:20.650178    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"]
Oct 31 01:28:22.885899 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:22.885858    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:28:32.886512 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:32.886465    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:28:35.670631 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:35.670588    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main" containerID="cri-o://bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d" gracePeriod=45
Oct 31 01:28:36.784147 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:36.784113    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 01:28:42.885908 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:42.885866    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:28:42.886409 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:42.885991    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:28:48.552631 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:28:48.552594    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:28:51.806593 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:51.806471    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerName="main" containerID="cri-o://a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f" gracePeriod=45
Oct 31 01:28:52.469443 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.469421    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:28:52.530718 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.530686    4888 generic.go:356] "Generic (PLEG): container finished" podID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerID="a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f" exitCode=0
Oct 31 01:28:52.530882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.530743    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 31 01:28:52.530882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.530771    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"0cc9247f-ffe9-412e-ab71-65e0d3fd924c","Type":"ContainerDied","Data":"a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f"}
Oct 31 01:28:52.530957 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.530878    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"0cc9247f-ffe9-412e-ab71-65e0d3fd924c","Type":"ContainerDied","Data":"ae2dea799bf12c329d6a9b4eb0d499d4e3871824bebb5ebc7f06407f68d51d03"}
Oct 31 01:28:52.530957 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.530910    4888 scope.go:122] "RemoveContainer" containerID="a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f"
Oct 31 01:28:52.540506 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.540489    4888 scope.go:122] "RemoveContainer" containerID="4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625"
Oct 31 01:28:52.548653 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.548637    4888 scope.go:122] "RemoveContainer" containerID="a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f"
Oct 31 01:28:52.548965 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:28:52.548942    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f\": container with ID starting with a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f not found: ID does not exist" containerID="a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f"
Oct 31 01:28:52.549025 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.548979    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f"} err="failed to get container status \"a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f\": rpc error: code = NotFound desc = could not find container \"a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f\": container with ID starting with a2c2fff2e8858a2309a34b4c1032353b6e9748d2457d67376c8b7618258e3f0f not found: ID does not exist"
Oct 31 01:28:52.549025 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.549002    4888 scope.go:122] "RemoveContainer" containerID="4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625"
Oct 31 01:28:52.549280 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:28:52.549264    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625\": container with ID starting with 4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625 not found: ID does not exist" containerID="4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625"
Oct 31 01:28:52.549324 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.549288    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625"} err="failed to get container status \"4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625\": rpc error: code = NotFound desc = could not find container \"4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625\": container with ID starting with 4bd2cf0c2b50a1c6f7de7d8d1b7cbdbb478a24ed0ccd027f292b638c0139d625 not found: ID does not exist"
Oct 31 01:28:52.582834 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582760    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-dshm\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.582834 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582806    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kube-api-access-zwdz7\" (UniqueName: \"kubernetes.io/projected/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kube-api-access-zwdz7\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.582834 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582829    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-home\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-home\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.583045 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582854    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-cabundle-cert\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.583045 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582876    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tls-certs\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.583045 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582899    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kserve-provision-location\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.583045 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582946    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-model-cache\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.583045 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.582973    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tmp-dir\") pod \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\" (UID: \"0cc9247f-ffe9-412e-ab71-65e0d3fd924c\") "
Oct 31 01:28:52.583970 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.583769    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-home" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:52.584121 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.583984    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-cabundle-cert" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:28:52.584274 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.584250    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-model-cache" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:52.585571 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.585547    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kube-api-access-zwdz7" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "kube-api-access-zwdz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:28:52.585754 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.585730    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-dshm" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:52.586088 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.586065    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tls-certs" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:28:52.601277 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.601240    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tmp-dir" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:52.611432 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.611401    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kserve-provision-location" pod "0cc9247f-ffe9-412e-ab71-65e0d3fd924c" (UID: "0cc9247f-ffe9-412e-ab71-65e0d3fd924c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:28:52.683727 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683698    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.683727 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683729    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwdz7\" (UniqueName: \"kubernetes.io/projected/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kube-api-access-zwdz7\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.683917 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683743    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.683917 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683755    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.683917 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683767    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.683917 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683779    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.683917 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683791    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.683917 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.683802    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc9247f-ffe9-412e-ab71-65e0d3fd924c-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:28:52.886904 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.886869    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:28:52.949135 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.949108    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 01:28:52.954730 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:52.954709    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 31 01:28:53.554588 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:28:53.554549    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" path="/var/lib/kubelet/pods/0cc9247f-ffe9-412e-ab71-65e0d3fd924c/volumes"
Oct 31 01:29:02.886211 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:02.886167    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:29:05.911908 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:05.911880    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:29:05.982541 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:05.982511    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe220633-7e8e-4e64-9774-cd12deab16f5-tls-certs\" (UniqueName: \"kubernetes.io/secret/fe220633-7e8e-4e64-9774-cd12deab16f5-tls-certs\") pod \"fe220633-7e8e-4e64-9774-cd12deab16f5\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") "
Oct 31 01:29:05.984567 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:05.984547    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe220633-7e8e-4e64-9774-cd12deab16f5-tls-certs" pod "fe220633-7e8e-4e64-9774-cd12deab16f5" (UID: "fe220633-7e8e-4e64-9774-cd12deab16f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:29:06.083143 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.083112    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe220633-7e8e-4e64-9774-cd12deab16f5-kube-api-access-69gts\" (UniqueName: \"kubernetes.io/projected/fe220633-7e8e-4e64-9774-cd12deab16f5-kube-api-access-69gts\") pod \"fe220633-7e8e-4e64-9774-cd12deab16f5\" (UID: \"fe220633-7e8e-4e64-9774-cd12deab16f5\") "
Oct 31 01:29:06.083299 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.083288    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe220633-7e8e-4e64-9774-cd12deab16f5-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:29:06.085281 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.085252    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe220633-7e8e-4e64-9774-cd12deab16f5-kube-api-access-69gts" pod "fe220633-7e8e-4e64-9774-cd12deab16f5" (UID: "fe220633-7e8e-4e64-9774-cd12deab16f5"). InnerVolumeSpecName "kube-api-access-69gts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:29:06.184424 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.184396    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69gts\" (UniqueName: \"kubernetes.io/projected/fe220633-7e8e-4e64-9774-cd12deab16f5-kube-api-access-69gts\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:29:06.593105 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.593076    4888 generic.go:356] "Generic (PLEG): container finished" podID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerID="bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d" exitCode=0
Oct 31 01:29:06.593275 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.593114    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"
Oct 31 01:29:06.593275 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.593130    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" event={"ID":"fe220633-7e8e-4e64-9774-cd12deab16f5","Type":"ContainerDied","Data":"bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d"}
Oct 31 01:29:06.593275 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.593166    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b" event={"ID":"fe220633-7e8e-4e64-9774-cd12deab16f5","Type":"ContainerDied","Data":"c1dd6f5257d8eff124c8a34b601fddf40ec1086ae54047ae718827ffde1e6b03"}
Oct 31 01:29:06.593275 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.593185    4888 scope.go:122] "RemoveContainer" containerID="bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d"
Oct 31 01:29:06.601616 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.601599    4888 scope.go:122] "RemoveContainer" containerID="bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d"
Oct 31 01:29:06.601885 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:29:06.601862    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d\": container with ID starting with bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d not found: ID does not exist" containerID="bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d"
Oct 31 01:29:06.601934 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.601894    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d"} err="failed to get container status \"bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d\": rpc error: code = NotFound desc = could not find container \"bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d\": container with ID starting with bdcb993e8e63b2998eb22701a562279e02fb370d0d0c7aff218c355532f1365d not found: ID does not exist"
Oct 31 01:29:06.751320 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.751289    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"]
Oct 31 01:29:06.797893 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:06.797865    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-schew6g8b"]
Oct 31 01:29:07.554166 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:07.554133    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" path="/var/lib/kubelet/pods/fe220633-7e8e-4e64-9774-cd12deab16f5/volumes"
Oct 31 01:29:18.186965 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.186937    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"]
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187241    4888 cpu_manager.go:414] "containerMap: removing container" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187253    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187267    4888 cpu_manager.go:414] "containerMap: removing container" podUID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerName="storage-initializer"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187272    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerName="storage-initializer"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187282    4888 cpu_manager.go:414] "containerMap: removing container" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187287    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187300    4888 cpu_manager.go:414] "containerMap: removing container" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187304    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187310    4888 cpu_manager.go:414] "containerMap: removing container" podUID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerName="main"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187314    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerName="main"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187381    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="3651906b-d2d2-4e06-a8ce-8a6b0349c55b" containerName="istio-proxy"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187391    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="54c71c62-a915-4baa-b60f-a8ff61c56dbf" containerName="istio-proxy"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187398    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe220633-7e8e-4e64-9774-cd12deab16f5" containerName="main"
Oct 31 01:29:18.187429 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.187404    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc9247f-ffe9-412e-ab71-65e0d3fd924c" containerName="main"
Oct 31 01:29:18.190092 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.190077    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.241301 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.241274    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 31 01:29:18.273427 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273396    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-home\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.273571 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273442    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-model-cache\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.273571 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273469    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tls-certs\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.273645 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273589    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-cabundle-cert\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.273679 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273651    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tmp-dir\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.273713 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273698    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kserve-provision-location\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.273746 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273719    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7jm\" (UniqueName: \"kubernetes.io/projected/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kube-api-access-pd7jm\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.273812 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.273795    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-dshm\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.287646 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.287625    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 31 01:29:18.287916 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.287897    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 31 01:29:18.288077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.287912    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:29:18.288077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.287959    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 31 01:29:18.288077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.287962    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-dockercfg-ns2dt\""
Oct 31 01:29:18.288077 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.287974    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-b668w\""
Oct 31 01:29:18.289897 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.289874    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"]
Oct 31 01:29:18.375135 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375106    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-dshm\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375141    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-home\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375175    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-model-cache\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375201    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tls-certs\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375311 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375237    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-cabundle-cert\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375459    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tmp-dir\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375508    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kserve-provision-location\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375537    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7jm\" (UniqueName: \"kubernetes.io/projected/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kube-api-access-pd7jm\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375568    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-home\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375797 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375595    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-model-cache\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375865 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375816    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kserve-provision-location\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.375865 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375853    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tmp-dir\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.376016 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.375995    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-cabundle-cert\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.377504 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.377484    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-dshm\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.377764 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.377748    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tls-certs\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.435758 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.435729    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7jm\" (UniqueName: \"kubernetes.io/projected/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kube-api-access-pd7jm\") pod \"tls-verification-test-kserve-84774d9859-z49zl\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.506887 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.506863    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:18.561414 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:29:18.561253    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:29:18.944388 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:18.944278    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"]
Oct 31 01:29:18.951796 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:29:18.951761    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd30c36_c1ab_445e_b2e0_7a8bd1ad128c.slice/crio-f8ae9037e953a4e464a1225270e953e5866a63b1d5bbe130f8223651fc537ab7 WatchSource:0}: Error finding container f8ae9037e953a4e464a1225270e953e5866a63b1d5bbe130f8223651fc537ab7: Status 404 returned error can't find the container with id f8ae9037e953a4e464a1225270e953e5866a63b1d5bbe130f8223651fc537ab7
Oct 31 01:29:19.653939 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:19.653903    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerStarted","Data":"93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e"}
Oct 31 01:29:19.654376 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:19.653954    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerStarted","Data":"f8ae9037e953a4e464a1225270e953e5866a63b1d5bbe130f8223651fc537ab7"}
Oct 31 01:29:19.654376 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:19.653995    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:20.660230 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:20.660198    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerStarted","Data":"fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b"}
Oct 31 01:29:22.675559 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:22.675525    4888 generic.go:356] "Generic (PLEG): container finished" podID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerID="fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b" exitCode=0
Oct 31 01:29:22.675957 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:22.675638    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerDied","Data":"fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b"}
Oct 31 01:29:23.698257 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:23.698225    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerStarted","Data":"a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e"}
Oct 31 01:29:28.507956 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:28.507918    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:28.508378 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:28.507981    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:28.509266 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:28.509240    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:29:37.708714 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.708658    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podStartSLOduration=17.655959276 podStartE2EDuration="19.708642885s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.052683609s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:23.934316963 +0000 UTC m=+3572.489204402" watchObservedRunningTime="2026-10-31 01:29:37.708642885 +0000 UTC m=+3586.263530317"
Oct 31 01:29:37.709170 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.708990    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"]
Oct 31 01:29:37.712333 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.712305    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.756015 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.755980    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"]
Oct 31 01:29:37.758116 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.758099    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.801638 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801590    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwqtn\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-kube-api-access-qwqtn\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.801638 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801629    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.801950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801654    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.801950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801763    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.801950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801791    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.801950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801805    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.801950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801821    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.801950 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801945    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.802254 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.801971    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903370 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903319    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.903558 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903386    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903558 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903458    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903558 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903490    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903558 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903506    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903558 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903532    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.903558 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903558    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903673    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903709    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903741    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903769    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903810    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903819    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903830    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.903882 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903833    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.904294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903922    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.904294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.903996    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwqtn\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-kube-api-access-qwqtn\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.904294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.904016    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.904294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.904046    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj855\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-kube-api-access-hj855\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:37.904294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.904067    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.904294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.904070    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.904294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.904224    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.904695 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.904587    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.906013 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.905980    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.906527 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.906507    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.919787 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.919762    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"]
Oct 31 01:29:37.963777 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.963702    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:37.963915 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:37.963783    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwqtn\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-kube-api-access-qwqtn\") pod \"openshift-ai-inference-openshift-default-568c495856-5499q\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:38.005109 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005070    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005113    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005241    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005268    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005499 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005298    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005499 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005448    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj855\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-kube-api-access-hj855\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005607 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005510    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005607 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005557    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005607 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005582    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005755 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005675    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-certs\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005809 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005750    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005864 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005841    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-credential-socket\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005987 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005910    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.005987 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.005934    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-data\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.007819 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.007790    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-envoy\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.007937 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.007915    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.027176 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.027135    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:38.048844 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.048800    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"]
Oct 31 01:29:38.065682 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.065652    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj855\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-kube-api-access-hj855\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.065963 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.065941    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-token\") pod \"openshift-ai-inference-openshift-default-568c495856-xp7fd\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.073919 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.073840    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:38.458749 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.458717    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"]
Oct 31 01:29:38.461546 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.461522    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"]
Oct 31 01:29:38.466950 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:29:38.466921    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb382f9c8_753b_41b5_9caf_3f8b5024f21f.slice/crio-9991d587e9eee2d0090ee2f1e00a722f2ff4ead690a76cc034b55016656c17b1 WatchSource:0}: Error finding container 9991d587e9eee2d0090ee2f1e00a722f2ff4ead690a76cc034b55016656c17b1: Status 404 returned error can't find the container with id 9991d587e9eee2d0090ee2f1e00a722f2ff4ead690a76cc034b55016656c17b1
Oct 31 01:29:38.468908 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.468875    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:29:38.468988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.468941    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:29:38.468988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.468970    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:29:38.503797 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:29:38.503768    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4465d6c5_9e7f_49c2_8890_1429c62fdd25.slice/crio-27b63a50e1cb915299085e96534c732cb567676f406f0e397a00b658e5f7fbda WatchSource:0}: Error finding container 27b63a50e1cb915299085e96534c732cb567676f406f0e397a00b658e5f7fbda: Status 404 returned error can't find the container with id 27b63a50e1cb915299085e96534c732cb567676f406f0e397a00b658e5f7fbda
Oct 31 01:29:38.506699 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.506667    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:29:38.506784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.506736    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:29:38.506784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.506763    4888 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"}
Oct 31 01:29:38.507388 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.507360    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:29:38.526471 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.526447    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:29:38.780674 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.780545    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" event={"ID":"4465d6c5-9e7f-49c2-8890-1429c62fdd25","Type":"ContainerStarted","Data":"08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab"}
Oct 31 01:29:38.780674 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.780620    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" event={"ID":"4465d6c5-9e7f-49c2-8890-1429c62fdd25","Type":"ContainerStarted","Data":"27b63a50e1cb915299085e96534c732cb567676f406f0e397a00b658e5f7fbda"}
Oct 31 01:29:38.782782 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.782750    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" event={"ID":"b382f9c8-753b-41b5-9caf-3f8b5024f21f","Type":"ContainerStarted","Data":"a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8"}
Oct 31 01:29:38.782930 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.782812    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" event={"ID":"b382f9c8-753b-41b5-9caf-3f8b5024f21f","Type":"ContainerStarted","Data":"9991d587e9eee2d0090ee2f1e00a722f2ff4ead690a76cc034b55016656c17b1"}
Oct 31 01:29:38.900390 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:38.897846    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podStartSLOduration=1.897829091 podStartE2EDuration="1.897829091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:38.897140608 +0000 UTC m=+3587.452028047" watchObservedRunningTime="2026-10-31 01:29:38.897829091 +0000 UTC m=+3587.452716541"
Oct 31 01:29:39.027721 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:39.027685    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:39.043430 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:39.037516    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-5499q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:29:39.043430 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:39.037595    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 01:29:39.074784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:39.074749    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:39.086608 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:39.086558    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xp7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:29:39.086823 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:39.086648    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:29:40.029371 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:40.028815    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-5499q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:29:40.029371 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:40.028879    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 01:29:40.074903 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:40.074864    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xp7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:29:40.075079 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:40.074919    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:29:41.029203 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:41.028737    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-5499q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:29:41.029203 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:41.028798    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 01:29:41.074424 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:41.074384    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xp7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:29:41.074591 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:41.074453    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:29:42.032377 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:42.028758    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-5499q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:29:42.032377 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:42.028823    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 01:29:42.074707 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:42.074661    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xp7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:29:42.074900 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:42.074725    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:29:43.034375 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:43.028755    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-5499q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:29:43.034375 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:43.028822    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 01:29:43.074341 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:43.074301    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xp7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:29:43.074694 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:43.074653    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:29:44.033375 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:44.032760    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-5499q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 01:29:44.033375 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:44.032839    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 01:29:44.074883 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:44.074846    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-xp7fd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 31 01:29:44.075277 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:44.074916    4888 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 31 01:29:45.031999 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:45.031963    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:45.032274 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:45.032252    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:45.032915 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:45.032894    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:29:45.078694 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:45.078659    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:45.079183 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:45.078944    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:45.079746 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:45.079710    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:29:45.197575 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:45.197522    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podStartSLOduration=8.197508291 podStartE2EDuration="8.197508291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:29:39.022755776 +0000 UTC m=+3587.577643215" watchObservedRunningTime="2026-10-31 01:29:45.197508291 +0000 UTC m=+3593.752395729"
Oct 31 01:29:48.507903 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:48.507860    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:29:48.551529 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:29:48.551484    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:29:51.935576 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:51.935543    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:29:51.950634 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:51.950610    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:29:58.508200 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:29:58.508151    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:30:08.508143 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:30:08.508088    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:30:18.507625 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:30:18.507574    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:30:18.552180 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:30:18.552141    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:30:28.507931 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:30:28.507834    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:30:38.507956 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:30:38.507898    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:30:48.507626 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:30:48.507587    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:30:48.552713 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:30:48.552673    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:30:58.508011 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:30:58.507959    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:31:08.507836 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:08.507787    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8001/health\": dial tcp 10.134.0.75:8001: connect: connection refused"
Oct 31 01:31:18.517450 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:18.517415    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:31:18.530042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:18.530018    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:31:18.556470 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:31:18.556436    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:31:30.807389 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:30.807360    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"]
Oct 31 01:31:45.832436 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:45.832375    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main" containerID="cri-o://a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e" gracePeriod=45
Oct 31 01:31:46.374764 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.374718    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="llm-d-routing-sidecar" containerID="cri-o://93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e" gracePeriod=45
Oct 31 01:31:46.459874 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.459840    4888 generic.go:356] "Generic (PLEG): container finished" podID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerID="a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e" exitCode=0
Oct 31 01:31:46.459988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.459903    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerDied","Data":"a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e"}
Oct 31 01:31:46.617662 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.617638    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:31:46.751405 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751373    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-model-cache\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751405 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751408    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tmp-dir\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751618 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751481    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kube-api-access-pd7jm\" (UniqueName: \"kubernetes.io/projected/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kube-api-access-pd7jm\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751618 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751503    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-home\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-home\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751618 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751533    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tls-certs\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751618 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751552    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-dshm\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751618 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751610    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-cabundle-cert\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751880 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751639    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kserve-provision-location\") pod \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\" (UID: \"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c\") "
Oct 31 01:31:46.751984 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.751659    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-model-cache" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.752142 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.752113    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-cabundle-cert" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:31:46.752423 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.752374    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-home" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.753935 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.753906    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kube-api-access-pd7jm" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "kube-api-access-pd7jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:31:46.754152 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.754132    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-dshm" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.754298 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.754270    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tls-certs" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:31:46.764205 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.764171    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tmp-dir" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.775079 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.775056    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kserve-provision-location" pod "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" (UID: "2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:31:46.852610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852540    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.852610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852566    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.852610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852575    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.852610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852586    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.852610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852595    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.852610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852603    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.852610 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852612    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7jm\" (UniqueName: \"kubernetes.io/projected/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-kube-api-access-pd7jm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:46.853118 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:46.852620    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:31:47.471684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.471648    4888 generic.go:356] "Generic (PLEG): container finished" podID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerID="93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e" exitCode=0
Oct 31 01:31:47.471860 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.471716    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"
Oct 31 01:31:47.471860 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.471768    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerDied","Data":"93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e"}
Oct 31 01:31:47.471860 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.471851    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl" event={"ID":"2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c","Type":"ContainerDied","Data":"f8ae9037e953a4e464a1225270e953e5866a63b1d5bbe130f8223651fc537ab7"}
Oct 31 01:31:47.471964 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.471882    4888 scope.go:122] "RemoveContainer" containerID="a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e"
Oct 31 01:31:47.480894 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.480877    4888 scope.go:122] "RemoveContainer" containerID="fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b"
Oct 31 01:31:47.488517 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.488500    4888 scope.go:122] "RemoveContainer" containerID="93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e"
Oct 31 01:31:47.496038 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.496020    4888 scope.go:122] "RemoveContainer" containerID="a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e"
Oct 31 01:31:47.496291 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:31:47.496270    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e\": container with ID starting with a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e not found: ID does not exist" containerID="a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e"
Oct 31 01:31:47.496438 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.496301    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e"} err="failed to get container status \"a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e\": rpc error: code = NotFound desc = could not find container \"a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e\": container with ID starting with a8813f25f8623576f27808046f6cbaf5f60c9f89c183f717f1bc8c701821476e not found: ID does not exist"
Oct 31 01:31:47.496438 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.496325    4888 scope.go:122] "RemoveContainer" containerID="fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b"
Oct 31 01:31:47.496927 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:31:47.496909    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b\": container with ID starting with fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b not found: ID does not exist" containerID="fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b"
Oct 31 01:31:47.496979 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.496934    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b"} err="failed to get container status \"fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b\": rpc error: code = NotFound desc = could not find container \"fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b\": container with ID starting with fe0a261e778bcb3d78894a68d46feeb6d0d964a10057be19f2788f33d209741b not found: ID does not exist"
Oct 31 01:31:47.496979 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.496951    4888 scope.go:122] "RemoveContainer" containerID="93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e"
Oct 31 01:31:47.497186 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:31:47.497168    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e\": container with ID starting with 93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e not found: ID does not exist" containerID="93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e"
Oct 31 01:31:47.497237 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.497192    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e"} err="failed to get container status \"93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e\": rpc error: code = NotFound desc = could not find container \"93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e\": container with ID starting with 93dc5b09c7d56ddd62280cb7c692c7a8700d8afd632e0b071972b59c0bb94f6e not found: ID does not exist"
Oct 31 01:31:47.678873 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.678842    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"]
Oct 31 01:31:47.725713 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:47.725634    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-84774d9859-z49zl"]
Oct 31 01:31:48.550945 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:31:48.550910    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:31:49.553547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:31:49.553520    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" path="/var/lib/kubelet/pods/2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c/volumes"
Oct 31 01:32:18.551035 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:32:18.550996    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:32:25.626740 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.626700    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"]
Oct 31 01:32:25.628067 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628040    4888 cpu_manager.go:414] "containerMap: removing container" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main"
Oct 31 01:32:25.628231 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628218    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main"
Oct 31 01:32:25.628328 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628318    4888 cpu_manager.go:414] "containerMap: removing container" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="llm-d-routing-sidecar"
Oct 31 01:32:25.628424 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628412    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="llm-d-routing-sidecar"
Oct 31 01:32:25.628534 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628524    4888 cpu_manager.go:414] "containerMap: removing container" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="storage-initializer"
Oct 31 01:32:25.628599 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628590    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="storage-initializer"
Oct 31 01:32:25.628835 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628823    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="main"
Oct 31 01:32:25.628922 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.628913    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd30c36-c1ab-445e-b2e0-7a8bd1ad128c" containerName="llm-d-routing-sidecar"
Oct 31 01:32:25.632549 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.632529    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.658676 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.658648    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.658838 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.658699    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.658838 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.658780    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw6vx\" (UniqueName: \"kubernetes.io/projected/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-kube-api-access-vw6vx\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.658838 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.658816    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.659023 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.658844    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.659023 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.658956    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.685452 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.685383    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"]
Oct 31 01:32:25.730645 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.730615    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-r5nv8\""
Oct 31 01:32:25.730779 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.730672    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Oct 31 01:32:25.731169 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.731145    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Oct 31 01:32:25.731469 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.731442    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Oct 31 01:32:25.759993 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.759967    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760083 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760019    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760083 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760047    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw6vx\" (UniqueName: \"kubernetes.io/projected/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-kube-api-access-vw6vx\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760083 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760066    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760198 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760090    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760198 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760109    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760563 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760535    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tmp-dir\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760662 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760562    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-home\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.760662 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.760611    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-model-cache\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.762586 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.762563    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-dshm\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.762679 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.762665    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tls-certs\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.820309 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.820273    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw6vx\" (UniqueName: \"kubernetes.io/projected/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-kube-api-access-vw6vx\") pod \"rolling-upgrade-test-kserve-5d77dc844-8p6s6\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:25.945411 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:25.945381    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:26.151692 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.151660    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"]
Oct 31 01:32:26.156162 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.156140    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.207053 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.206986    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-j88m9\""
Oct 31 01:32:26.211914 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.211889    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"]
Oct 31 01:32:26.264602 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.264566    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bb8dce-f630-43c2-8c57-35ca1ffab178-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.264769 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.264642    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh5j5\" (UniqueName: \"kubernetes.io/projected/c4bb8dce-f630-43c2-8c57-35ca1ffab178-kube-api-access-xh5j5\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.335285 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.335248    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"]
Oct 31 01:32:26.342466 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:32:26.342437    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod882c43d9_c90d_4274_8bb4_bd42ae41f1b7.slice/crio-f06ca395cf07394193da49fc06dc9f24961a119c0c4172778f5786a67a0f93e8 WatchSource:0}: Error finding container f06ca395cf07394193da49fc06dc9f24961a119c0c4172778f5786a67a0f93e8: Status 404 returned error can't find the container with id f06ca395cf07394193da49fc06dc9f24961a119c0c4172778f5786a67a0f93e8
Oct 31 01:32:26.365524 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.365498    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bb8dce-f630-43c2-8c57-35ca1ffab178-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.365609 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.365546    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh5j5\" (UniqueName: \"kubernetes.io/projected/c4bb8dce-f630-43c2-8c57-35ca1ffab178-kube-api-access-xh5j5\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.367812 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.367795    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bb8dce-f630-43c2-8c57-35ca1ffab178-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.425855 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.425829    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh5j5\" (UniqueName: \"kubernetes.io/projected/c4bb8dce-f630-43c2-8c57-35ca1ffab178-kube-api-access-xh5j5\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.469544 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.469454    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:26.654600 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.654565    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" event={"ID":"882c43d9-c90d-4274-8bb4-bd42ae41f1b7","Type":"ContainerStarted","Data":"8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3"}
Oct 31 01:32:26.655099 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.654619    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" event={"ID":"882c43d9-c90d-4274-8bb4-bd42ae41f1b7","Type":"ContainerStarted","Data":"f06ca395cf07394193da49fc06dc9f24961a119c0c4172778f5786a67a0f93e8"}
Oct 31 01:32:26.770778 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.770674    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" podStartSLOduration=1.770658882 podStartE2EDuration="1.770658882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:26.76923495 +0000 UTC m=+3755.324122390" watchObservedRunningTime="2026-10-31 01:32:26.770658882 +0000 UTC m=+3755.325546371"
Oct 31 01:32:26.840459 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.840428    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"]
Oct 31 01:32:26.847781 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:32:26.847752    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0 WatchSource:0}: Error finding container 0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0: Status 404 returned error can't find the container with id 0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0
Oct 31 01:32:26.946387 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:26.946324    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:27.660748 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:27.660715    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" event={"ID":"c4bb8dce-f630-43c2-8c57-35ca1ffab178","Type":"ContainerStarted","Data":"82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502"}
Oct 31 01:32:27.661246 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:27.660766    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" event={"ID":"c4bb8dce-f630-43c2-8c57-35ca1ffab178","Type":"ContainerStarted","Data":"0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0"}
Oct 31 01:32:27.661246 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:27.660795    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:32:27.776156 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:27.776103    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" podStartSLOduration=1.7760890360000001 podStartE2EDuration="1.776089036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:32:27.77408109 +0000 UTC m=+3756.328968545" watchObservedRunningTime="2026-10-31 01:32:27.776089036 +0000 UTC m=+3756.330976468"
Oct 31 01:32:35.946542 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:35.946500    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:35.959130 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:35.959103    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:36.714843 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:36.714817    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:32:48.551319 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:32:48.551285    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:32:58.667209 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:32:58.667178    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:33:08.357073 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.357042    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"]
Oct 31 01:33:08.371517 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.371481    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"]
Oct 31 01:33:08.380402 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:33:08.380332    4888 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3" cmd=["/bin/sleep","15"]
Oct 31 01:33:08.380535 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.380425    4888 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 31 01:33:08.380535 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:33:08.380442    4888 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" podUID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" containerName="main" containerID="cri-o://8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3"
Oct 31 01:33:08.380535 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.380513    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" podUID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" containerName="main" containerID="cri-o://8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3" gracePeriod=60
Oct 31 01:33:08.667362 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.667309    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:33:08.723566 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.723546    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:33:08.820599 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820568    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-model-cache\") pod \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") "
Oct 31 01:33:08.820784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820619    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tls-certs\" (UniqueName: \"kubernetes.io/secret/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tls-certs\") pod \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") "
Oct 31 01:33:08.820784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820640    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tmp-dir\") pod \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") "
Oct 31 01:33:08.820784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820668    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-home\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-home\") pod \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") "
Oct 31 01:33:08.820784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820704    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-kube-api-access-vw6vx\" (UniqueName: \"kubernetes.io/projected/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-kube-api-access-vw6vx\") pod \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") "
Oct 31 01:33:08.820784 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820721    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-dshm\") pod \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\" (UID: \"882c43d9-c90d-4274-8bb4-bd42ae41f1b7\") "
Oct 31 01:33:08.821042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820902    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-model-cache" pod "882c43d9-c90d-4274-8bb4-bd42ae41f1b7" (UID: "882c43d9-c90d-4274-8bb4-bd42ae41f1b7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:08.821042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820963    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-home" pod "882c43d9-c90d-4274-8bb4-bd42ae41f1b7" (UID: "882c43d9-c90d-4274-8bb4-bd42ae41f1b7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:08.821042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.820979    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tmp-dir" pod "882c43d9-c90d-4274-8bb4-bd42ae41f1b7" (UID: "882c43d9-c90d-4274-8bb4-bd42ae41f1b7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:08.821042 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.821031    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:08.823066 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.823037    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-kube-api-access-vw6vx" pod "882c43d9-c90d-4274-8bb4-bd42ae41f1b7" (UID: "882c43d9-c90d-4274-8bb4-bd42ae41f1b7"). InnerVolumeSpecName "kube-api-access-vw6vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:08.823178 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.823074    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tls-certs" pod "882c43d9-c90d-4274-8bb4-bd42ae41f1b7" (UID: "882c43d9-c90d-4274-8bb4-bd42ae41f1b7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:08.823178 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.823116    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-dshm" pod "882c43d9-c90d-4274-8bb4-bd42ae41f1b7" (UID: "882c43d9-c90d-4274-8bb4-bd42ae41f1b7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:33:08.864375 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.864323    4888 generic.go:356] "Generic (PLEG): container finished" podID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" containerID="8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3" exitCode=0
Oct 31 01:33:08.864520 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.864400    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"
Oct 31 01:33:08.864520 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.864403    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" event={"ID":"882c43d9-c90d-4274-8bb4-bd42ae41f1b7","Type":"ContainerDied","Data":"8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3"}
Oct 31 01:33:08.864520 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.864518    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" event={"ID":"882c43d9-c90d-4274-8bb4-bd42ae41f1b7","Type":"ContainerDied","Data":"f06ca395cf07394193da49fc06dc9f24961a119c0c4172778f5786a67a0f93e8"}
Oct 31 01:33:08.864622 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.864542    4888 scope.go:122] "RemoveContainer" containerID="8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3"
Oct 31 01:33:08.873413 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.873395    4888 scope.go:122] "RemoveContainer" containerID="8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3"
Oct 31 01:33:08.873686 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:33:08.873663    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3\": container with ID starting with 8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3 not found: ID does not exist" containerID="8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3"
Oct 31 01:33:08.873796 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.873692    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3"} err="failed to get container status \"8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3\": rpc error: code = NotFound desc = could not find container \"8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3\": container with ID starting with 8d4fbe632cc7b3422ed22eaa623c27723621910a6c53cf6bf76ca37b5901aeb3 not found: ID does not exist"
Oct 31 01:33:08.921684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.921623    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:08.921684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.921646    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw6vx\" (UniqueName: \"kubernetes.io/projected/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-kube-api-access-vw6vx\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:08.921684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.921655    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:08.921684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.921663    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:08.921684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:08.921671    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/882c43d9-c90d-4274-8bb4-bd42ae41f1b7-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:09.023697 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:09.023666    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"]
Oct 31 01:33:09.030270 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:09.030246    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6"]
Oct 31 01:33:09.554160 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:09.554126    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" path="/var/lib/kubelet/pods/882c43d9-c90d-4274-8bb4-bd42ae41f1b7/volumes"
Oct 31 01:33:09.705508 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:09.705470    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-5d77dc844-8p6s6" podUID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:33:15.141990 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.141955    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"]
Oct 31 01:33:15.142439 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.142419    4888 cpu_manager.go:414] "containerMap: removing container" podUID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" containerName="main"
Oct 31 01:33:15.142489 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.142443    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" containerName="main"
Oct 31 01:33:15.142537 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.142526    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="882c43d9-c90d-4274-8bb4-bd42ae41f1b7" containerName="main"
Oct 31 01:33:15.145704 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.145686    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.197364 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.197320    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 31 01:33:15.197504 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.197488    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 31 01:33:15.243130 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.243103    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 31 01:33:15.243294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.243151    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 31 01:33:15.243294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.243161    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 31 01:33:15.243294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.243221    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4r5wb\""
Oct 31 01:33:15.245292 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.245273    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"]
Oct 31 01:33:15.276396 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276367    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-model-cache\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.276396 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276398    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tmp-dir\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.276582 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276481    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-home\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.276582 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276537    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-cabundle-cert\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.276582 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276568    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-dshm\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.276767 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276586    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kserve-provision-location\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.276767 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276665    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tls-certs\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.276767 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.276703    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99qrg\" (UniqueName: \"kubernetes.io/projected/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kube-api-access-99qrg\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377136 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377110    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-dshm\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377136 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377140    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kserve-provision-location\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377175    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tls-certs\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377203    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99qrg\" (UniqueName: \"kubernetes.io/projected/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kube-api-access-99qrg\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377247    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-model-cache\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377271    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tmp-dir\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377323    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-home\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377611 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377376    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-cabundle-cert\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377763 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377737    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-home\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377763 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377753    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-model-cache\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377973 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377806    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tmp-dir\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.377973 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.377916    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kserve-provision-location\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.378365 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.378324    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-cabundle-cert\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.379599 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.379574    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-dshm\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.379904 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.379888    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tls-certs\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.437877 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.437817    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99qrg\" (UniqueName: \"kubernetes.io/projected/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kube-api-access-99qrg\") pod \"prestop-hook-test-kserve-6578f4856b-6njzk\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.460679 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.460653    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:15.887661 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:15.887629    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"]
Oct 31 01:33:15.894858 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:33:15.894829    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e73f3d3_6b84_4fb9_a670_f9a95a6ac748.slice/crio-6a7cc0cd453757b14492e4dbd42c5bc1bcb99c047e707ef0cb22fd8d74bcc88c WatchSource:0}: Error finding container 6a7cc0cd453757b14492e4dbd42c5bc1bcb99c047e707ef0cb22fd8d74bcc88c: Status 404 returned error can't find the container with id 6a7cc0cd453757b14492e4dbd42c5bc1bcb99c047e707ef0cb22fd8d74bcc88c
Oct 31 01:33:16.908292 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:16.908256    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" event={"ID":"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748","Type":"ContainerStarted","Data":"641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe"}
Oct 31 01:33:16.908729 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:16.908335    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" event={"ID":"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748","Type":"ContainerStarted","Data":"6a7cc0cd453757b14492e4dbd42c5bc1bcb99c047e707ef0cb22fd8d74bcc88c"}
Oct 31 01:33:18.551971 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:33:18.551937    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:33:18.667003 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:18.666968    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:33:18.920446 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:18.920330    4888 generic.go:356] "Generic (PLEG): container finished" podID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerID="641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe" exitCode=0
Oct 31 01:33:18.920446 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:18.920402    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" event={"ID":"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748","Type":"ContainerDied","Data":"641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe"}
Oct 31 01:33:19.928068 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:19.928032    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" event={"ID":"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748","Type":"ContainerStarted","Data":"25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5"}
Oct 31 01:33:20.141149 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:20.141082    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" podStartSLOduration=3.070714136 podStartE2EDuration="5.141061165s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.070347029s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:33:20.093178903 +0000 UTC m=+3808.648066376" watchObservedRunningTime="2026-10-31 01:33:20.141061165 +0000 UTC m=+3808.695948603"
Oct 31 01:33:20.461148 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:20.461071    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:23.382098 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:23.382063    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main" containerID="cri-o://82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502" gracePeriod=45
Oct 31 01:33:25.461188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:25.461151    4888 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:33:25.462853 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:25.462809    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 31 01:33:28.667155 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:28.667115    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:33:28.667546 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:28.667244    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:33:35.461802 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:35.461757    4888 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": dial tcp 10.134.0.80:8000: connect: connection refused"
Oct 31 01:33:35.557119 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:35.557085    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"]
Oct 31 01:33:38.666568 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:38.666530    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:33:48.552367 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:33:48.552308    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:33:48.667016 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:48.666976    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:33:50.580396 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:50.580317    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="main" containerID="cri-o://25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5" gracePeriod=45
Oct 31 01:33:53.632201 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:53.632174    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:33:53.702175 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:53.702149    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4bb8dce-f630-43c2-8c57-35ca1ffab178-kube-api-access-xh5j5\" (UniqueName: \"kubernetes.io/projected/c4bb8dce-f630-43c2-8c57-35ca1ffab178-kube-api-access-xh5j5\") pod \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") "
Oct 31 01:33:53.702329 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:53.702201    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4bb8dce-f630-43c2-8c57-35ca1ffab178-tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bb8dce-f630-43c2-8c57-35ca1ffab178-tls-certs\") pod \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\" (UID: \"c4bb8dce-f630-43c2-8c57-35ca1ffab178\") "
Oct 31 01:33:53.704328 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:53.704303    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4bb8dce-f630-43c2-8c57-35ca1ffab178-tls-certs" pod "c4bb8dce-f630-43c2-8c57-35ca1ffab178" (UID: "c4bb8dce-f630-43c2-8c57-35ca1ffab178"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:33:53.704328 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:53.704314    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4bb8dce-f630-43c2-8c57-35ca1ffab178-kube-api-access-xh5j5" pod "c4bb8dce-f630-43c2-8c57-35ca1ffab178" (UID: "c4bb8dce-f630-43c2-8c57-35ca1ffab178"). InnerVolumeSpecName "kube-api-access-xh5j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:33:53.803118 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:53.803083    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh5j5\" (UniqueName: \"kubernetes.io/projected/c4bb8dce-f630-43c2-8c57-35ca1ffab178-kube-api-access-xh5j5\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:53.803118 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:53.803112    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4bb8dce-f630-43c2-8c57-35ca1ffab178-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:33:54.101294 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.101210    4888 generic.go:356] "Generic (PLEG): container finished" podID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerID="82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502" exitCode=0
Oct 31 01:33:54.101476 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.101288    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"
Oct 31 01:33:54.101476 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.101298    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" event={"ID":"c4bb8dce-f630-43c2-8c57-35ca1ffab178","Type":"ContainerDied","Data":"82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502"}
Oct 31 01:33:54.101476 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.101426    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49" event={"ID":"c4bb8dce-f630-43c2-8c57-35ca1ffab178","Type":"ContainerDied","Data":"0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0"}
Oct 31 01:33:54.101476 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.101452    4888 scope.go:122] "RemoveContainer" containerID="82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502"
Oct 31 01:33:54.110143 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.110118    4888 scope.go:122] "RemoveContainer" containerID="82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502"
Oct 31 01:33:54.110506 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:33:54.110487    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502\": container with ID starting with 82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502 not found: ID does not exist" containerID="82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502"
Oct 31 01:33:54.110584 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.110516    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502"} err="failed to get container status \"82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502\": rpc error: code = NotFound desc = could not find container \"82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502\": container with ID starting with 82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502 not found: ID does not exist"
Oct 31 01:33:54.261684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.261646    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"]
Oct 31 01:33:54.308467 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:54.308437    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49"]
Oct 31 01:33:55.553845 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:33:55.553809    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" path="/var/lib/kubelet/pods/c4bb8dce-f630-43c2-8c57-35ca1ffab178/volumes"
Oct 31 01:34:18.551505 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:18.551463    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:34:26.898489 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:26.898452    4888 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/a906cd25a02eef469b31f33a2fba217afe1de4a1b7029e99a96fc2fa7a150813/diff" to get inode usage: stat /var/lib/containers/storage/overlay/a906cd25a02eef469b31f33a2fba217afe1de4a1b7029e99a96fc2fa7a150813/diff: no such file or directory, extraDiskErr: could not stat "/var/log/pods/e2e-test-rolling-upgrade-coordination-a9229638_rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49_c4bb8dce-f630-43c2-8c57-35ca1ffab178/main/0.log" to get inode usage: stat /var/log/pods/e2e-test-rolling-upgrade-coordination-a9229638_rolling-upgrade-test-kserve-router-scheduler-7455cbc656-27p49_c4bb8dce-f630-43c2-8c57-35ca1ffab178/main/0.log: no such file or directory
Oct 31 01:34:35.601870 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:35.601821    4888 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e73f3d3_6b84_4fb9_a670_f9a95a6ac748.slice/crio-6a7cc0cd453757b14492e4dbd42c5bc1bcb99c047e707ef0cb22fd8d74bcc88c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-conmon-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:34:35.602313 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:35.601869    4888 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-conmon-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 01:34:35.602313 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:35.601935    4888 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-conmon-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:34:35.602313 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:35.601941    4888 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e73f3d3_6b84_4fb9_a670_f9a95a6ac748.slice/crio-6a7cc0cd453757b14492e4dbd42c5bc1bcb99c047e707ef0cb22fd8d74bcc88c\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-conmon-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 01:34:35.606203 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:35.606163    4888 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-0faafe38852d6df5c2ade1a57b80e53e0f1190576342c529a4f2ead2c993c6c0\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-conmon-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4bb8dce_f630_43c2_8c57_35ca1ffab178.slice/crio-82927fb0d024ad85ac8a2395c9055f130307e9bfa2da36cf9b66cba4b50a6502.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 01:34:36.231328 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.231305    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6578f4856b-6njzk_7e73f3d3-6b84-4fb9-a670-f9a95a6ac748/main/0.log"
Oct 31 01:34:36.232997 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.232982    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:34:36.281422 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281400    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tmp-dir\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281542 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281460    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-dshm\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281542 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281497    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-home\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-home\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281661 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281551    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-cabundle-cert\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281661 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281585    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kserve-provision-location\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281661 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281636    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tls-certs\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281803 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281675    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-model-cache\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281803 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281712    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kube-api-access-99qrg\" (UniqueName: \"kubernetes.io/projected/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kube-api-access-99qrg\") pod \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\" (UID: \"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748\") "
Oct 31 01:34:36.281953 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281920    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-home" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:36.282049 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.281952    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-cabundle-cert" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:36.282142 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.282124    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tmp-dir" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:36.282473 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.282443    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-model-cache" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:36.284240 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.284220    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kube-api-access-99qrg" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "kube-api-access-99qrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:36.284435 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.284411    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-dshm" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:36.284547 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.284525    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tls-certs" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:34:36.305249 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.305227    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-6578f4856b-6njzk_7e73f3d3-6b84-4fb9-a670-f9a95a6ac748/main/0.log"
Oct 31 01:34:36.305565 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.305546    4888 generic.go:356] "Generic (PLEG): container finished" podID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerID="25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5" exitCode=137
Oct 31 01:34:36.305655 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.305594    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" event={"ID":"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748","Type":"ContainerDied","Data":"25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5"}
Oct 31 01:34:36.305655 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.305609    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"
Oct 31 01:34:36.305655 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.305632    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk" event={"ID":"7e73f3d3-6b84-4fb9-a670-f9a95a6ac748","Type":"ContainerDied","Data":"6a7cc0cd453757b14492e4dbd42c5bc1bcb99c047e707ef0cb22fd8d74bcc88c"}
Oct 31 01:34:36.305655 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.305654    4888 scope.go:122] "RemoveContainer" containerID="25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5"
Oct 31 01:34:36.305929 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.305908    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kserve-provision-location" pod "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" (UID: "7e73f3d3-6b84-4fb9-a670-f9a95a6ac748"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:36.314788 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.314771    4888 scope.go:122] "RemoveContainer" containerID="641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe"
Oct 31 01:34:36.322310 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.322291    4888 scope.go:122] "RemoveContainer" containerID="25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5"
Oct 31 01:34:36.322580 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:36.322559    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5\": container with ID starting with 25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5 not found: ID does not exist" containerID="25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5"
Oct 31 01:34:36.322631 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.322590    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5"} err="failed to get container status \"25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5\": rpc error: code = NotFound desc = could not find container \"25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5\": container with ID starting with 25bad36ce929edf1dd89ea5ffe1f3ab4f164f4fec9b4cab43538cd5a83f877b5 not found: ID does not exist"
Oct 31 01:34:36.322631 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.322610    4888 scope.go:122] "RemoveContainer" containerID="641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe"
Oct 31 01:34:36.322841 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:36.322818    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe\": container with ID starting with 641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe not found: ID does not exist" containerID="641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe"
Oct 31 01:34:36.322891 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.322846    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe"} err="failed to get container status \"641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe\": rpc error: code = NotFound desc = could not find container \"641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe\": container with ID starting with 641ebed2638042da3058265db02e7d278e5969ecdade868a66b7475b894aa4fe not found: ID does not exist"
Oct 31 01:34:36.383344 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383284    4888 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-cabundle-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.383344 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383306    4888 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kserve-provision-location\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.383344 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383316    4888 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tls-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.383344 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383325    4888 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-model-cache\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.383344 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383334    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99qrg\" (UniqueName: \"kubernetes.io/projected/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-kube-api-access-99qrg\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.383344 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383343    4888 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-tmp-dir\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.383600 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383368    4888 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-dshm\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.383600 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.383375    4888 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748-home\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:36.726876 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.723784    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"]
Oct 31 01:34:36.770823 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:36.770792    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-6578f4856b-6njzk"]
Oct 31 01:34:37.553073 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:37.553035    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" path="/var/lib/kubelet/pods/7e73f3d3-6b84-4fb9-a670-f9a95a6ac748/volumes"
Oct 31 01:34:48.551733 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:48.551701    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:34:51.966607 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:51.966529    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:34:51.983221 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:51.983197    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-5jpqn_3d584253-d14a-48e9-bfe6-563acb954e1f/csi-node-driver-registrar/1.log"
Oct 31 01:34:52.685548 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:52.685508    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"]
Oct 31 01:34:52.685788 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:52.685740    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy" containerID="cri-o://08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab" gracePeriod=30
Oct 31 01:34:52.732446 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:52.732403    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"]
Oct 31 01:34:52.732670 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:52.732646    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy" containerID="cri-o://a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8" gracePeriod=30
Oct 31 01:34:52.734004 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:52.733986    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"]
Oct 31 01:34:52.734183 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:52.734153    4888 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy" containerID="cri-o://8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9" gracePeriod=30
Oct 31 01:34:57.326489 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:57.326452    4888 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-568c495856-jtzw2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:34:57.326881 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:57.326527    4888 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:34:57.984243 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:57.984216    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:34:58.015790 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.015768    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:34:58.018663 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.018643    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:34:58.054676 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054646    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-podinfo\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.054805 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054683    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-credential-socket\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.054805 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054705    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-certs\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.054805 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054725    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-kube-api-access-hj855\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-kube-api-access-hj855\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.054805 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054757    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-data\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.054805 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054793    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-envoy\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.055071 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054844    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istiod-ca-cert\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.055071 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054898    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-token\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-token\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.055071 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054935    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-socket\") pod \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\" (UID: \"4465d6c5-9e7f-49c2-8890-1429c62fdd25\") "
Oct 31 01:34:58.055071 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.055005    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-certs" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.055071 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.054992    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-credential-socket" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.055335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.055080    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-data" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.055335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.055255    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.055335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.055265    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-socket" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.055335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.055277    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.055335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.055291    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.055335 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.055291    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istiod-ca-cert" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:58.057041 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.057015    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-kube-api-access-hj855" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "kube-api-access-hj855". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:58.057041 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.057027    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-token" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:58.057175 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.057049    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-podinfo" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:58.057175 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.057048    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-envoy" pod "4465d6c5-9e7f-49c2-8890-1429c62fdd25" (UID: "4465d6c5-9e7f-49c2-8890-1429c62fdd25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.156188 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156157    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-socket\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156336 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156204    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istiod-ca-cert\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156336 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156236    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-credential-socket\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156336 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156262    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-podinfo\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156336 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156306    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-token\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-token\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156341    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-kube-api-access-qwqtn\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-kube-api-access-qwqtn\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156388    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-podinfo\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156419    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-certs\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156461    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-socket" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.156579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156488    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-socket\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156512    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-credential-socket" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.156579 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156557    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-credential-socket\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156588    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-envoy\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156630    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-data\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156657    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istiod-ca-cert" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156664    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-certs\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156696    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-token\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-token\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156723    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istiod-ca-cert\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156750    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-data\") pod \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\" (UID: \"b382f9c8-753b-41b5-9caf-3f8b5024f21f\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156778    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-envoy\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156804    4888 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-kube-api-access-sss99\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-kube-api-access-sss99\") pod \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\" (UID: \"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7\") "
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156943    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-certs" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.156988 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.156957    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-data" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157101    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157120    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157505    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157530    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157545    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157558    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157573    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157586    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj855\" (UniqueName: \"kubernetes.io/projected/4465d6c5-9e7f-49c2-8890-1429c62fdd25-kube-api-access-hj855\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157600    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157614    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.157736 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157628    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4465d6c5-9e7f-49c2-8890-1429c62fdd25-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.158291 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157757    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-credential-socket" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.158291 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.157963    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-socket" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.158291 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.158007    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-data" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.158291 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.158191    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-certs" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.158540 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.158313    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istiod-ca-cert" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:34:58.159105 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.159043    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-kube-api-access-qwqtn" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "kube-api-access-qwqtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:58.159105 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.159072    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-podinfo" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:58.159891 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.159870    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-kube-api-access-sss99" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "kube-api-access-sss99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:58.159979 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.159897    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-token" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:58.160478 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.160458    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-podinfo" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:34:58.160586 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.160476    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-envoy" pod "b382f9c8-753b-41b5-9caf-3f8b5024f21f" (UID: "b382f9c8-753b-41b5-9caf-3f8b5024f21f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.161186 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.161167    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-token" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:34:58.161419 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.161395    4888 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-envoy" pod "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" (UID: "bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258540    4888 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-credential-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258570    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258579    4888 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-certs\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258588    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258601    4888 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istiod-ca-cert\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258612    4888 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-data\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258619    4888 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-envoy\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258623 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258627    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sss99\" (UniqueName: \"kubernetes.io/projected/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-kube-api-access-sss99\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258637    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258647    4888 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-token\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258654    4888 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwqtn\" (UniqueName: \"kubernetes.io/projected/b382f9c8-753b-41b5-9caf-3f8b5024f21f-kube-api-access-qwqtn\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258663    4888 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b382f9c8-753b-41b5-9caf-3f8b5024f21f-istio-podinfo\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.258985 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.258674    4888 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7-workload-socket\") on node \"ip-10-0-131-97.us-west-2.compute.internal\" DevicePath \"\""
Oct 31 01:34:58.406919 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.406875    4888 generic.go:356] "Generic (PLEG): container finished" podID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerID="8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9" exitCode=0
Oct 31 01:34:58.406919 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.406926    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"
Oct 31 01:34:58.407507 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.406959    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" event={"ID":"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7","Type":"ContainerDied","Data":"8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9"}
Oct 31 01:34:58.407507 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.407017    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2" event={"ID":"bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7","Type":"ContainerDied","Data":"fd7905e77499c97e985970375490b843b293562eca72cd227a3465177db61a87"}
Oct 31 01:34:58.407507 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.407045    4888 scope.go:122] "RemoveContainer" containerID="8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9"
Oct 31 01:34:58.408539 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.408502    4888 generic.go:356] "Generic (PLEG): container finished" podID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerID="08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab" exitCode=0
Oct 31 01:34:58.408676 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.408557    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" event={"ID":"4465d6c5-9e7f-49c2-8890-1429c62fdd25","Type":"ContainerDied","Data":"08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab"}
Oct 31 01:34:58.408676 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.408568    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"
Oct 31 01:34:58.408676 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.408620    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd" event={"ID":"4465d6c5-9e7f-49c2-8890-1429c62fdd25","Type":"ContainerDied","Data":"27b63a50e1cb915299085e96534c732cb567676f406f0e397a00b658e5f7fbda"}
Oct 31 01:34:58.410324 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.410300    4888 generic.go:356] "Generic (PLEG): container finished" podID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerID="a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8" exitCode=0
Oct 31 01:34:58.410589 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.410393    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" event={"ID":"b382f9c8-753b-41b5-9caf-3f8b5024f21f","Type":"ContainerDied","Data":"a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8"}
Oct 31 01:34:58.410589 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.410439    4888 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"
Oct 31 01:34:58.410589 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.410437    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q" event={"ID":"b382f9c8-753b-41b5-9caf-3f8b5024f21f","Type":"ContainerDied","Data":"9991d587e9eee2d0090ee2f1e00a722f2ff4ead690a76cc034b55016656c17b1"}
Oct 31 01:34:58.417461 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.417443    4888 scope.go:122] "RemoveContainer" containerID="8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9"
Oct 31 01:34:58.417758 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:58.417731    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9\": container with ID starting with 8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9 not found: ID does not exist" containerID="8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9"
Oct 31 01:34:58.417854 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.417767    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9"} err="failed to get container status \"8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9\": rpc error: code = NotFound desc = could not find container \"8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9\": container with ID starting with 8e4d0339877c0dc2672319896f690c933b33ceb81d9ffe9225d4c349b74ed3f9 not found: ID does not exist"
Oct 31 01:34:58.417854 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.417790    4888 scope.go:122] "RemoveContainer" containerID="08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab"
Oct 31 01:34:58.426385 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.426364    4888 scope.go:122] "RemoveContainer" containerID="08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab"
Oct 31 01:34:58.426626 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:58.426608    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab\": container with ID starting with 08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab not found: ID does not exist" containerID="08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab"
Oct 31 01:34:58.426684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.426633    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab"} err="failed to get container status \"08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab\": rpc error: code = NotFound desc = could not find container \"08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab\": container with ID starting with 08c34eb148de8276743af36f589ae862408775a008252f0e763419efab2f06ab not found: ID does not exist"
Oct 31 01:34:58.426684 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.426649    4888 scope.go:122] "RemoveContainer" containerID="a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8"
Oct 31 01:34:58.434250 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.434234    4888 scope.go:122] "RemoveContainer" containerID="a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8"
Oct 31 01:34:58.434496 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:34:58.434480    4888 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8\": container with ID starting with a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8 not found: ID does not exist" containerID="a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8"
Oct 31 01:34:58.434543 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.434501    4888 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8"} err="failed to get container status \"a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8\": rpc error: code = NotFound desc = could not find container \"a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8\": container with ID starting with a72a0b1e62a60a45b2f6c3af48d48ab82b03d926c690000d0b7cd15a948d0cb8 not found: ID does not exist"
Oct 31 01:34:58.568279 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.568204    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"]
Oct 31 01:34:58.574063 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.574039    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-jtzw2"]
Oct 31 01:34:58.681078 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.681049    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"]
Oct 31 01:34:58.686948 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.686924    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-xp7fd"]
Oct 31 01:34:58.794969 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.794939    4888 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"]
Oct 31 01:34:58.800312 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:58.800288    4888 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-568c495856-5499q"]
Oct 31 01:34:59.553747 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:59.553716    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" path="/var/lib/kubelet/pods/4465d6c5-9e7f-49c2-8890-1429c62fdd25/volumes"
Oct 31 01:34:59.554131 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:59.554105    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" path="/var/lib/kubelet/pods/b382f9c8-753b-41b5-9caf-3f8b5024f21f/volumes"
Oct 31 01:34:59.554473 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:34:59.554460    4888 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" path="/var/lib/kubelet/pods/bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7/volumes"
Oct 31 01:35:18.550918 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:35:18.550882    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:35:48.551944 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:35:48.551901    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:35:54.902646 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:35:54.902609    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-47szr_81b60237-5b3f-4d48-a1df-b1fa8a211dd4/discovery/0.log"
Oct 31 01:35:56.401530 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:35:56.401502    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-xc7wn_8d60c10e-c079-4f3b-a08e-843d51c9d457/istio-proxy/0.log"
Oct 31 01:35:56.403583 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:35:56.403563    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-xc7wn_8d60c10e-c079-4f3b-a08e-843d51c9d457/istio-proxy/1.log"
Oct 31 01:35:59.650898 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:35:59.650866    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-47szr_81b60237-5b3f-4d48-a1df-b1fa8a211dd4/discovery/0.log"
Oct 31 01:36:00.956140 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:00.956112    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-xc7wn_8d60c10e-c079-4f3b-a08e-843d51c9d457/istio-proxy/0.log"
Oct 31 01:36:00.957895 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:00.957882    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-568c495856-xc7wn_8d60c10e-c079-4f3b-a08e-843d51c9d457/istio-proxy/1.log"
Oct 31 01:36:04.049175 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:04.049142    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-557c8f8584-vvnzr_eb687462-eacd-481c-a7a6-bf64ea2e3d29/authorino/0.log"
Oct 31 01:36:04.413865 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:04.413789    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-66498fdf6c-m4lqp_c0a4397b-047b-40b6-8b72-acaca7e6d2af/manager/0.log"
Oct 31 01:36:05.462498 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:05.462465    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-69b7bb8c5c-wpvmf_fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa/manager/0.log"
Oct 31 01:36:10.610815 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.610778    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjbw9/must-gather-94ndx"]
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611077    4888 cpu_manager.go:414] "containerMap: removing container" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611088    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611099    4888 cpu_manager.go:414] "containerMap: removing container" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="storage-initializer"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611105    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="storage-initializer"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611117    4888 cpu_manager.go:414] "containerMap: removing container" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611122    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611131    4888 cpu_manager.go:414] "containerMap: removing container" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="main"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611135    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="main"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611145    4888 cpu_manager.go:414] "containerMap: removing container" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611149    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611159    4888 cpu_manager.go:414] "containerMap: removing container" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611163    4888 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611210    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4bb8dce-f630-43c2-8c57-35ca1ffab178" containerName="main"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611216    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e73f3d3-6b84-4fb9-a670-f9a95a6ac748" containerName="main"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611223    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1cbb4b-7cc6-4dd1-908f-6c3d353ce8b7" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611229    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="4465d6c5-9e7f-49c2-8890-1429c62fdd25" containerName="istio-proxy"
Oct 31 01:36:10.611286 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.611235    4888 memory_manager.go:365] "RemoveStaleState removing state" podUID="b382f9c8-753b-41b5-9caf-3f8b5024f21f" containerName="istio-proxy"
Oct 31 01:36:10.615500 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.615482    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:10.669405 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.669376    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjbw9/must-gather-94ndx"]
Oct 31 01:36:10.709671 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.709643    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f06af5b5-6886-4745-9460-484807d2cb26-must-gather-output\") pod \"must-gather-94ndx\" (UID: \"f06af5b5-6886-4745-9460-484807d2cb26\") " pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:10.709802 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.709677    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z7sp\" (UniqueName: \"kubernetes.io/projected/f06af5b5-6886-4745-9460-484807d2cb26-kube-api-access-8z7sp\") pod \"must-gather-94ndx\" (UID: \"f06af5b5-6886-4745-9460-484807d2cb26\") " pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:10.714605 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.714579    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjbw9\"/\"kube-root-ca.crt\""
Oct 31 01:36:10.714605 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.714579    4888 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sjbw9\"/\"openshift-service-ca.crt\""
Oct 31 01:36:10.714720 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.714586    4888 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sjbw9\"/\"default-dockercfg-jfw5t\""
Oct 31 01:36:10.810362 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.810330    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f06af5b5-6886-4745-9460-484807d2cb26-must-gather-output\") pod \"must-gather-94ndx\" (UID: \"f06af5b5-6886-4745-9460-484807d2cb26\") " pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:10.810512 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.810396    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z7sp\" (UniqueName: \"kubernetes.io/projected/f06af5b5-6886-4745-9460-484807d2cb26-kube-api-access-8z7sp\") pod \"must-gather-94ndx\" (UID: \"f06af5b5-6886-4745-9460-484807d2cb26\") " pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:10.810704 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.810684    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f06af5b5-6886-4745-9460-484807d2cb26-must-gather-output\") pod \"must-gather-94ndx\" (UID: \"f06af5b5-6886-4745-9460-484807d2cb26\") " pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:10.870694 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.870629    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z7sp\" (UniqueName: \"kubernetes.io/projected/f06af5b5-6886-4745-9460-484807d2cb26-kube-api-access-8z7sp\") pod \"must-gather-94ndx\" (UID: \"f06af5b5-6886-4745-9460-484807d2cb26\") " pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:10.928439 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:10.928407    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjbw9/must-gather-94ndx"
Oct 31 01:36:11.299363 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:11.299317    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjbw9/must-gather-94ndx"]
Oct 31 01:36:11.307206 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:36:11.307176    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf06af5b5_6886_4745_9460_484807d2cb26.slice/crio-3a2fd5cb47974f807005cc8b75c9302f0bdb610c15f700e4b1b21d9a0acdedd4 WatchSource:0}: Error finding container 3a2fd5cb47974f807005cc8b75c9302f0bdb610c15f700e4b1b21d9a0acdedd4: Status 404 returned error can't find the container with id 3a2fd5cb47974f807005cc8b75c9302f0bdb610c15f700e4b1b21d9a0acdedd4
Oct 31 01:36:11.308981 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:11.308962    4888 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:36:11.732340 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:11.732305    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjbw9/must-gather-94ndx" event={"ID":"f06af5b5-6886-4745-9460-484807d2cb26","Type":"ContainerStarted","Data":"3a2fd5cb47974f807005cc8b75c9302f0bdb610c15f700e4b1b21d9a0acdedd4"}
Oct 31 01:36:13.744751 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:13.744717    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjbw9/must-gather-94ndx" event={"ID":"f06af5b5-6886-4745-9460-484807d2cb26","Type":"ContainerStarted","Data":"de15537d026f096a60a921732df2274a364c0523fce75b40eac3db5a70234628"}
Oct 31 01:36:13.745145 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:13.744761    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjbw9/must-gather-94ndx" event={"ID":"f06af5b5-6886-4745-9460-484807d2cb26","Type":"ContainerStarted","Data":"5ee7df957686a37c56260daff1e41ee39c915130e1391bad3c3596ecffe41cdd"}
Oct 31 01:36:13.860480 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:13.860421    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-sjbw9/must-gather-94ndx" podStartSLOduration=2.381834329 podStartE2EDuration="3.86040742s" totalImagesPullingTime="1.478573091s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:13.859413753 +0000 UTC m=+3982.414301187" watchObservedRunningTime="2026-10-31 01:36:13.86040742 +0000 UTC m=+3982.415294858"
Oct 31 01:36:18.552215 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:36:18.552107    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"
Oct 31 01:36:19.841291 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:19.841261    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hrxqv_2a1760d4-777c-4853-bbbf-0a9afae8f45f/global-pull-secret-syncer/0.log"
Oct 31 01:36:23.137164 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:23.137131    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gmcv9_7b07fd43-f514-4533-9523-0f8f97930639/konnectivity-agent/0.log"
Oct 31 01:36:23.947712 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:23.947647    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-97.us-west-2.compute.internal_0b932d77d38edb2300c0d4d58dae2752/haproxy/0.log"
Oct 31 01:36:29.248413 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:29.248379    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-557c8f8584-vvnzr_eb687462-eacd-481c-a7a6-bf64ea2e3d29/authorino/0.log"
Oct 31 01:36:29.620890 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:29.620802    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-66498fdf6c-m4lqp_c0a4397b-047b-40b6-8b72-acaca7e6d2af/manager/0.log"
Oct 31 01:36:31.689261 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:31.689227    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-69b7bb8c5c-wpvmf_fc5fdab6-2b4e-41f9-9d83-25cd01acd3fa/manager/0.log"
Oct 31 01:36:32.736959 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.736857    4888 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"]
Oct 31 01:36:32.742941 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.742914    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.800733 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.800701    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-proc\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.800935 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.800753    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffsc5\" (UniqueName: \"kubernetes.io/projected/ef68b995-d704-45ea-aef6-f5b087e0bef6-kube-api-access-ffsc5\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.800935 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.800847    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-sys\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.800935 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.800867    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-lib-modules\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.801066 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.800949    4888 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-podres\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.824903 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.824868    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"]
Oct 31 01:36:32.901563 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901530    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-proc\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901574    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffsc5\" (UniqueName: \"kubernetes.io/projected/ef68b995-d704-45ea-aef6-f5b087e0bef6-kube-api-access-ffsc5\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901597    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-sys\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901611    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-lib-modules\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901650    4888 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-podres\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901674    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-proc\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901678    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-sys\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901727    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-lib-modules\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.901762 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.901759    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef68b995-d704-45ea-aef6-f5b087e0bef6-podres\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:32.962297 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:32.961937    4888 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffsc5\" (UniqueName: \"kubernetes.io/projected/ef68b995-d704-45ea-aef6-f5b087e0bef6-kube-api-access-ffsc5\") pod \"perf-node-gather-daemonset-5qzvm\" (UID: \"ef68b995-d704-45ea-aef6-f5b087e0bef6\") " pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:33.058383 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:33.058280    4888 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:33.432664 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:33.432565    4888 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"]
Oct 31 01:36:33.442100 ip-10-0-131-97 kubenswrapper[4888]: W1031 01:36:33.442070    4888 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef68b995_d704_45ea_aef6_f5b087e0bef6.slice/crio-e0cb169d4f0f52a065d590691cc910fffa4397b54cf00bd4ae1474ae4fab672a WatchSource:0}: Error finding container e0cb169d4f0f52a065d590691cc910fffa4397b54cf00bd4ae1474ae4fab672a: Status 404 returned error can't find the container with id e0cb169d4f0f52a065d590691cc910fffa4397b54cf00bd4ae1474ae4fab672a
Oct 31 01:36:33.868033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:33.865860    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm" event={"ID":"ef68b995-d704-45ea-aef6-f5b087e0bef6","Type":"ContainerStarted","Data":"bb544ee82e381b00ae0d07ad5a692f7b5270b13c3686310668ce11f6a77eb558"}
Oct 31 01:36:33.868033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:33.865930    4888 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm" event={"ID":"ef68b995-d704-45ea-aef6-f5b087e0bef6","Type":"ContainerStarted","Data":"e0cb169d4f0f52a065d590691cc910fffa4397b54cf00bd4ae1474ae4fab672a"}
Oct 31 01:36:33.868033 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:33.866750    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:33.978932 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:33.978876    4888 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm" podStartSLOduration=1.9788631639999998 podStartE2EDuration="1.978863164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:36:33.977740053 +0000 UTC m=+4002.532627493" watchObservedRunningTime="2026-10-31 01:36:33.978863164 +0000 UTC m=+4002.533750603"
Oct 31 01:36:34.168538 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:34.168474    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9d3522d-c03d-466e-b816-25caa6905168/alertmanager/0.log"
Oct 31 01:36:34.582224 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:34.582198    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9d3522d-c03d-466e-b816-25caa6905168/config-reloader/0.log"
Oct 31 01:36:34.994000 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:34.993973    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9d3522d-c03d-466e-b816-25caa6905168/kube-rbac-proxy-web/0.log"
Oct 31 01:36:35.406529 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:35.406462    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9d3522d-c03d-466e-b816-25caa6905168/kube-rbac-proxy/0.log"
Oct 31 01:36:35.819490 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:35.819464    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9d3522d-c03d-466e-b816-25caa6905168/kube-rbac-proxy-metric/0.log"
Oct 31 01:36:36.231593 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:36.231558    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9d3522d-c03d-466e-b816-25caa6905168/prom-label-proxy/0.log"
Oct 31 01:36:36.642731 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:36.642659    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a9d3522d-c03d-466e-b816-25caa6905168/init-config-reloader/0.log"
Oct 31 01:36:39.541656 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:39.541626    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cnsr_02ae9325-803d-4acc-bafe-4349f1548558/node-exporter/0.log"
Oct 31 01:36:39.951981 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:39.951953    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cnsr_02ae9325-803d-4acc-bafe-4349f1548558/kube-rbac-proxy/0.log"
Oct 31 01:36:40.362895 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:40.362801    4888 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9cnsr_02ae9325-803d-4acc-bafe-4349f1548558/init-textfile/0.log"
Oct 31 01:36:40.886395 ip-10-0-131-97 kubenswrapper[4888]: I1031 01:36:40.886340    4888 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-sjbw9/perf-node-gather-daemonset-5qzvm"
Oct 31 01:36:48.551311 ip-10-0-131-97 kubenswrapper[4888]: E1031 01:36:48.551273    4888 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nm5ww" containerName="extractor"