Oct 30 18:36:12.894696 ip-10-0-129-88 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:13.300658 ip-10-0-129-88 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:13.300658 ip-10-0-129-88 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:13.300658 ip-10-0-129-88 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:13.300658 ip-10-0-129-88 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:13.307181 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307094    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.307181 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307177    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.307181 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307181    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.307181 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307184    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.307181 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307187    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.307181 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307190    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307193    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307196    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307199    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307202    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307204    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307207    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307210    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307212    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307215    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307218    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307220    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307224    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307226    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307229    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307232    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307235    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307238    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307240    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307243    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.307402 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307246    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307249    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307251    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307254    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307256    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307259    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307267    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307270    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307272    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307275    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307277    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307280    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307283    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307285    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307289    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307292    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307294    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307298    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307302    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.307960 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307305    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307308    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307311    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307313    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307316    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307319    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307321    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307324    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307326    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307329    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307332    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307334    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307337    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307340    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307342    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307345    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307348    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307351    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307353    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307357    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.308409 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307359    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307362    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307364    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307367    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307369    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307372    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307374    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307377    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307380    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307383    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307385    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307388    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307390    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307393    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307397    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307401    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307405    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307408    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307411    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.308909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307415    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307418    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307421    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307423    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307439    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307442    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307445    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307447    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307450    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307453    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307456    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307458    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307461    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307463    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307466    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307469    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307472    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307476    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307479    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.309358 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307482    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307485    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307488    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.307491    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308266    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308274    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308277    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308280    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308283    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308285    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308288    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308291    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308293    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308296    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308300    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308303    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308305    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308308    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308311    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.309841 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308313    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308316    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308318    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308321    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308324    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308326    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308329    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308331    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308334    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308336    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308339    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308342    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308344    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308348    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308351    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308353    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308356    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308359    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308362    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308364    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.310298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308367    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308369    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308383    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308387    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308391    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308394    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308396    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308399    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308402    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308405    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308408    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308410    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308413    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308416    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308418    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308421    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308424    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308441    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308446    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308450    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.310815 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308454    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308457    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308460    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308464    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308468    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308471    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308475    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308478    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308481    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308484    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308487    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308490    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308492    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308495    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308497    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308500    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308509    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308513    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308516    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.311293 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308519    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308524    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308528    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308531    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308534    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308537    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308541    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308544    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308546    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308549    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308552    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308555    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308558    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308561    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308563    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308566    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308568    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308571    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308574    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.311762 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308577    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308580    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308582    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308585    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308588    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308590    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308593    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308595    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308598    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308600    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308603    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308605    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.308608    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309104    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309109    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309113    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309116    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309119    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309121    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.312215 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309126    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309130    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309133    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309135    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309138    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309141    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309144    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309146    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309149    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309152    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309154    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309157    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309160    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309163    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309166    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309168    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309171    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309174    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309176    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309179    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.312686 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309182    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309184    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309187    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309189    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309192    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309194    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309197    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309199    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309203    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309206    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309210    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309214    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309217    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309220    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309223    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309226    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309229    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309232    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309235    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.313170 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309238    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309240    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309243    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309246    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309249    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309252    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309254    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309257    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309259    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309262    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309264    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309267    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309270    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309272    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309275    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309277    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309280    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309282    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309284    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.313653 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309287    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309290    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309293    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309295    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309299    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309301    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309304    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309307    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309309    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309312    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309314    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309317    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309319    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309321    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309324    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309326    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309329    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309332    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309334    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.314091 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309337    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309339    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309342    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309344    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309347    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309349    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309352    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309355    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309358    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309360    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309364    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309366    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309369    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309372    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309375    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309377    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309379    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309382    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309385    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309387    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.314579 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309390    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309393    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.309396    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310580    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310590    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310597    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310602    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310606    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310610    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310614    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310618    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310621    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310624    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310628    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310632    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310635    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310638    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310641    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310644    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310647    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310650    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310654    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310656    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:13.315105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310659    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310663    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310666    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310670    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310673    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310676    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310680    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310683    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310687    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310690    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310693    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310696    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310701    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310704    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310707    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310710    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310713    2574 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310716    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310721    2574 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310724    2574 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310727    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310730    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310733    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310737    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:13.315659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310740    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310743    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310746    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310749    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310756    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310759    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310762    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310765    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310768    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310772    2574 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310776    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310779    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310783    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310786    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310789    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310792    2574 flags.go:64] FLAG: --help="false"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310802    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310807    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310810    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310813    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310817    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310820    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310823    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:13.316273 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310827    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310830    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310833    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310836    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310839    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310843    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310846    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310849    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310852    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310855    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310857    2574 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310861    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310864    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310869    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310874    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310877    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310880    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310884    2574 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310888    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310892    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310895    2574 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310898    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310902    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310905    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310909    2574 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:13.316831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310912    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310915    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310919    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310922    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310925    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310928    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310931    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310938    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310941    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310944    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310948    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310950    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310954    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310957    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310960    2574 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310963    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310966    2574 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0586cbc82d324dc74"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310969    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310972    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310975    2574 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310979    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310986    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310989    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310992    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.310995    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311000    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:13.317505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311003    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311006    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311009    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311012    2574 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311016    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311019    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311022    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311025    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311028    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311032    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311035    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311039    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311041    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311044    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311048    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311050    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311053    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311056    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311059    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311065    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311068    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311071    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311075    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311078    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311081    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:13.318118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311084    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311088    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311091    2574 flags.go:64] FLAG: --v="2"
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311096    2574 flags.go:64] FLAG: --version="false"
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311100    2574 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311104    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311109    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311259    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311263    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311267    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311270    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311273    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311276    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311279    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311282    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311284    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311287    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311291    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311294    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311297    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.318733 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311300    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311303    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311305    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311308    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311311    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311313    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311316    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311319    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311321    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311324    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311327    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311329    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311332    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311336    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311339    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311341    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311344    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311346    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311351    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311353    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.319225 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311356    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311359    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311361    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311364    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311367    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311370    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311372    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311375    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311377    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311380    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311389    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311392    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311394    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311397    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311399    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311402    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311405    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311407    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311410    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311412    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.319757 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311415    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311418    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311420    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311423    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311436    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311441    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311444    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311446    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311449    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311451    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311455    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311458    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311461    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311463    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311466    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311469    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311472    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311474    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311477    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.320230 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311479    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311482    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311485    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311487    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311491    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311494    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311497    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311500    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311502    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311505    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311509    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311514    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311517    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311520    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311523    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311526    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311529    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311531    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311535    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.320691 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311538    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311540    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311543    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311545    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311549    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311552    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311555    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311557    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311560    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311563    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311567    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311571    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311574    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311577    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311580    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.311607    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:13.321143 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311659    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311665    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311668    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311672    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311674    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311677    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311680    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311682    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311685    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311687    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311690    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311692    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311695    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311698    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311701    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311703    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311707    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311709    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311712    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311715    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.321549 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311719    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311721    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311724    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311726    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311729    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311732    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311734    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311737    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311739    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311742    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311745    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311748    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311750    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311753    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311756    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311759    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311761    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311764    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311767    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311769    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.322027 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311772    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311775    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311777    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311780    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311782    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311785    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311788    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311790    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311794    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311796    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311799    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311802    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311805    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311807    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311810    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311812    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311815    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311817    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311820    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.322527 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311823    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311825    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311828    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311830    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311833    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311835    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311837    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311840    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311843    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311846    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311849    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311852    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311854    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311857    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311861    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311865    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311868    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311870    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311873    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.322988 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311875    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311878    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311881    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311883    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311886    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311888    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311891    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311894    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311896    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311899    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311901    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311904    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311906    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311909    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311913    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311917    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311920    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311923    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311925    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.323457 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311928    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311930    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311933    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311935    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311938    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311941    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311943    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311946    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.311948    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.319414    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.319444    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.319460    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.319464    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319523    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319529    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319533    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319536    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319539    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319542    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.323935 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319545    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319548    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319550    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319553    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319556    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319558    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319561    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319564    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319566    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319569    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319572    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319574    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319577    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319580    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319582    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319585    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319590    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319595    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319599    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319602    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.324379 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319605    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319607    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319610    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319613    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319616    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319619    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319621    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319624    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319626    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319629    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319632    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319634    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319637    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319640    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319642    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319645    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319647    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319650    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319652    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319655    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.324866 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319657    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319660    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319663    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319667    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319671    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319676    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319679    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319681    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319684    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319687    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319690    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319692    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319695    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319698    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319700    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319705    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319709    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319712    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319715    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.325367 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319718    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319720    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319723    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319725    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319728    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319730    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319733    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319735    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319738    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319742    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319747    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319751    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319755    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319758    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319761    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319764    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319767    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319770    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319772    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319775    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.325832 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319778    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319780    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319783    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319785    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319790    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319793    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319797    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319799    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319802    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319805    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319807    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319810    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319812    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319815    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319819    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319823    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319827    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319831    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319834    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.326298 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319837    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.319839    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.319867    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320011    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320017    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320020    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320023    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320025    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320028    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320031    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320033    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320036    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320038    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320041    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320043    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320046    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.326766 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320050    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320055    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320060    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320062    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320065    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320068    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320070    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320073    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320075    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320078    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320081    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320083    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320086    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320088    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320091    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320093    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320096    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320098    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320101    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.327152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320104    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320106    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320109    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320111    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320114    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320117    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320121    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320126    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320133    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320138    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320141    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320144    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320146    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320149    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320151    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320154    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320157    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320160    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320163    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.327616 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320165    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320168    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320170    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320173    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320175    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320178    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320181    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320183    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320186    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320188    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320191    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320193    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320195    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320198    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320201    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320206    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320210    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320214    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320217    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320220    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.328054 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320222    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320225    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320228    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320230    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320233    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320236    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320238    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320242    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320246    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320249    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320253    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320256    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320258    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320261    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320264    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320267    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320270    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320272    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320278    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320283    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.328717 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320287    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320291    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320294    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320297    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320299    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320302    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320304    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320307    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320310    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320312    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320315    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320317    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320320    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320323    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320325    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.320350    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:13.329185 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320384    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320389    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320393    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320396    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320398    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320401    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320403    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320407    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320409    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320412    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320415    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320417    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320420    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320423    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320439    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320446    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320449    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320452    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320455    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:13.329573 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320457    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320460    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320463    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320466    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320470    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320474    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320477    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320479    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320482    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320485    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320488    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320491    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320493    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320496    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320498    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320501    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320504    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320508    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320512    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:13.330003 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320517    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320521    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320524    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320527    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320530    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320532    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320535    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320537    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320540    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320544    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320546    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320549    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320551    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320554    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320556    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320559    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320562    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320564    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320567    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320569    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:13.330455 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320572    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320576    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320579    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320581    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320584    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320590    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320596    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320601    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320603    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320606    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320609    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320611    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320614    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320616    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320619    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320623    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320625    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320628    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320630    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320633    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:13.330933 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320636    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320640    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320642    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320645    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320647    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320650    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320653    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320657    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320659    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320663    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320667    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320672    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320676    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320679    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320682    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320685    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320687    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320690    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320693    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320695    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:13.332152 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320698    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320701    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320704    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320706    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320709    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320711    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320714    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:13.320716    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.320770    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.324008    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.325099    2574 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.325188    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.325220    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:13.333006 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.328721    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:13.334749 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.334726    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:13.344980 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.344963    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:13.348340 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.348320    2574 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:13.349108 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.349082    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:13.351931 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.351909    2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 879b331d-b122-4565-a679-eb9df7d20076:/dev/nvme0n1p3 b90366e3-eb25-410e-aeb2-6e7fb747acfe:/dev/nvme0n1p4]
Oct 30 18:36:13.351983 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.351930    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 18:36:13.357242 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.357135    2574 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:13.355615038 +0000 UTC m=+0.360399344 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3042064 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2c0f6017d1ba85cb50e341645233 SystemUUID:ec2a2c0f-6017-d1ba-85cb-50e341645233 BootID:19716777-e17c-43e3-984f-f02f1117aa59 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:df:b0:b4:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:df:b0:b4:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:8d:6c:d1:46:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:13.357242 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.357241    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:13.357356 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.357345    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:13.360791 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.360766    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:13.360915 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.360794    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-88.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:13.360962 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.360916    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:13.360962 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.360925    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:13.360962 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.360957    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:13.361763 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.361753    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:13.362941 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.362930    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:13.367641 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.367629    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:13.367768 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.367758    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:13.368759 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.368750    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:13.368797 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.368763    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:13.368797 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.368775    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:13.368797 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.368784    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:13.368797 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.368791    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:13.369087 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.369077    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:13.371364 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.371348    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:13.377546 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.377533    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:13.378383 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378372    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:13.378419 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378401    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:13.378625 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378612    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:13.378660 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378634    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:13.378660 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378641    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:13.378660 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378651    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:13.378660 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378659    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:13.378757 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.378674    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:13.379291 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.379283    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:13.379321 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.379292    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:13.379321 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.379299    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:13.379321 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.379305    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:13.379321 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.379318    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:13.379453 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.379327    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:13.380085 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.380077    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:13.380117 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.380086    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:13.383867 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.383856    2574 server.go:1263] "Started kubelet"
Oct 30 18:36:13.383935 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.383900    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:13.384330 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.384147    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:13.384711 ip-10-0-129-88 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:13.389542 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.389520    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:13.391156 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.391134    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:13.392100 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.392076    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 18:36:13.392201 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.392115    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:13.398069 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.398047    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:13.398771 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.398754    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:13.398885 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.398867    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:13.399583 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399566    2574 factory.go:55] Registering systemd factory
Oct 30 18:36:13.399583 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399583    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:13.399754 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399720    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:13.399754 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399701    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:13.399754 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399749    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:13.400000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399774    2574 factory.go:153] Registering CRI-O factory
Oct 30 18:36:13.400000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399788    2574 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:13.400000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399872    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:13.400000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399893    2574 factory.go:103] Registering Raw factory
Oct 30 18:36:13.400000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399906    2574 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:13.400000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399909    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:13.400000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.399915    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:13.400991 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.400976    2574 manager.go:319] Starting recovery of all containers
Oct 30 18:36:13.412798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.412048    2574 manager.go:324] Recovery completed
Oct 30 18:36:13.416027 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.416015    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.417603 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.417569    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.417690 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.417608    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.417690 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.417623    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.418085 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.418071    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:13.418085 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.418081    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.418199 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.418083    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:13.418199 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.418122    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:13.418761 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.418745    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.418843 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.418777    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.418843 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.418792    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.420393 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.420376    2574 policy_none.go:50] "Start"
Oct 30 18:36:13.420458 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.420399    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:13.420458 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.420413    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:13.422516 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.422504    2574 policy_none.go:44] "Start"
Oct 30 18:36:13.466028 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.465993    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:13.468472 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.468457    2574 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:13.468554 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.468496    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:13.468554 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.468508    2574 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:13.468745 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.468733    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:13.468796 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.468744    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:13.468866 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.468850    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:13.468981 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.468965    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:13.469046 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.468990    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:13.469412 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.469387    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:13.469504 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.469469    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:13.484463 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.484424    2574 container_manager_linux.go:682] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2607 into the system container \"/system.slice\": <nil>"
Oct 30 18:36:13.518234 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.518194    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-88.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 18:36:13.518234 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.518198    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-88.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 18:36:13.519456 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.518210    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.us-west-2.compute.internal.18e3627ecdeca8ef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:13.383829743 +0000 UTC m=+0.388614047,LastTimestamp:2026-10-30 18:36:13.383829743 +0000 UTC m=+0.388614047,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.us-west-2.compute.internal,}"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.568871    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.569941    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.569978    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.569990    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.570016    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.579014    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.580159    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.580179    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.580198    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:13.581080 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.580231    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 18:36:13.585994 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.585972    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 18:36:13.585994 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.585977    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 18:36:13.586121 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.585927    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.us-west-2.compute.internal.18e3627ecfefd441  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-88.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:13.417591873 +0000 UTC m=+0.422376179,LastTimestamp:2026-10-30 18:36:13.417591873 +0000 UTC m=+0.422376179,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.us-west-2.compute.internal,}"
Oct 30 18:36:13.586466 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.586449    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-88.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 18:36:13.630382 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.630358    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:36:13.630494 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.630422    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hfs7q"
Oct 30 18:36:13.666020 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.665939    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-88.us-west-2.compute.internal.18e3627ecff0303c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-88.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:13.41761542 +0000 UTC m=+0.422399727,LastTimestamp:2026-10-30 18:36:13.41761542 +0000 UTC m=+0.422399727,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.us-west-2.compute.internal,}"
Oct 30 18:36:13.680618 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.680597    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"]
Oct 30 18:36:13.680690 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.680650    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.681466 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.681450    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.681517 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.681485    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.681517 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.681507    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.683693 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.683680    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.683852 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.683836    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.683933 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.683873    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.684360 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.684338    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.684360 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.684348    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.684491 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.684368    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.684491 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.684380    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.684491 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.684368    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.684491 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.684453    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.686519 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.686505    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.686594 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.686528    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.687196 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.687169    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.687196 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.687195    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.687341 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.687214    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.701315 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.701293    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82007190ff1dd353d8af659976a136ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"82007190ff1dd353d8af659976a136ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.701387 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.701317    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.701387 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.701335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.715023 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.715002    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.717410 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.717394    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.733079 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.733060    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:13.734183 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.734169    2574 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-129-88.us-west-2.compute.internal\" not found" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.802144 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.802121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82007190ff1dd353d8af659976a136ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"82007190ff1dd353d8af659976a136ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.802239 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.802147    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.802239 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.802171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.802311 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.802239    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.802311 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.802269    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.802311 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.802239    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82007190ff1dd353d8af659976a136ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"82007190ff1dd353d8af659976a136ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.827144 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.827125    2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-88.us-west-2.compute.internal\" not found" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.935005 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.934932    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:13.935843 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.935827    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:13.935911 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.935856    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:13.935911 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.935868    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:13.935911 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.935893    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.991393 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:13.991369    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:13.991393 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:13.991394    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-88.us-west-2.compute.internal\": node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:14.019894 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:14.019873    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:14.023008 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:14.022994    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:14.110440 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:14.110261    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:15.719731 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:15.719666    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:15.820245 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:15.820219    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:15.833423 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:15.833394    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 18:36:15.833522 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:15.833506    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 18:36:15.833654 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:15.833582    2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a000931687ba14c969afc60d3cadd041-7b43774db3a78fc8.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-129-88.us-west-2.compute.internal.18e3627ecff0303c\": read tcp 10.0.129.88:43608->18.227.231.83:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-129-88.us-west-2.compute.internal.18e3627ecff0303c  default   4702 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-88.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-88.us-west-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-88.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 18:36:13 +0000 UTC,LastTimestamp:2026-10-30 18:36:13.681501339 +0000 UTC m=+0.686285644,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-88.us-west-2.compute.internal,}"
Oct 30 18:36:15.920893 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:15.920868    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:16.021748 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:16.021688    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:16.067003 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.066984    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:16.067917 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.067900    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:16.068001 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.067930    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:16.068001 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.067940    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:16.074570 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.074553    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:16.122144 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:16.122123    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:16.144713 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.144669    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:31:15 +0000 UTC" deadline="2028-08-06 18:54:02.1000456 +0000 UTC"
Oct 30 18:36:16.144713 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.144708    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15504h17m45.955339207s"
Oct 30 18:36:16.222866 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:16.222836    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:16.323915 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:16.323852    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:16.389946 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.389918    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:16.407896 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.407873    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:16.476392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.476356    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:16.478223 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.478208    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:16.537029 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.537003    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:16.634835 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:16.634803    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82007190ff1dd353d8af659976a136ef.slice/crio-0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32 WatchSource:0}: Error finding container 0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32: Status 404 returned error can't find the container with id 0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32
Oct 30 18:36:16.635377 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:16.635357    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be7e0652d2c7a6c3979448764fb16b6.slice/crio-7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797 WatchSource:0}: Error finding container 7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797: Status 404 returned error can't find the container with id 7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797
Oct 30 18:36:16.640881 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.640858    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:16.706435 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.706414    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:16.879692 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.879616    2574 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:16.907769 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.907752    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:16.986987 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.986964    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:16.987276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.987259    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5b6w7","kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal","openshift-dns/node-resolver-s6975","openshift-image-registry/node-ca-zfktp","openshift-multus/multus-2kfph","openshift-network-diagnostics/network-check-target-hktmd","openshift-ovn-kubernetes/ovnkube-node-5lk89","openshift-cluster-node-tuning-operator/tuned-m8v44","openshift-multus/multus-additional-cni-plugins-m4dhb","openshift-multus/network-metrics-daemon-fv5vj","openshift-network-operator/iptables-alerter-fcsph","kube-system/global-pull-secret-syncer-k8tk8"]
Oct 30 18:36:16.991108 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.991090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:16.993131 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.993114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:16.995720 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.995706    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:16.997995 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:16.997979    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:16.998064 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:16.998049    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:17.000231 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.000215    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.002537 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.002521    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.004856 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.004834    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.007042 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.007027    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.009219 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.009204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:17.009310 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.009261    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:17.012841 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.012812    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.014715 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.014697    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.017124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.017102    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:17.017191 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.017173    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:17.021391 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021372    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.021484 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-tmp\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.021484 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-var-lib-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.021484 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021425    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-env-overrides\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.021484 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021440    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-socket-dir-parent\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.021642 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021493    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-tuned\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.021642 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddgrz\" (UniqueName: \"kubernetes.io/projected/19d1dc0f-27db-4784-916c-dbf422a4d602-kube-api-access-ddgrz\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.021642 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021587    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d82m\" (UniqueName: \"kubernetes.io/projected/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kube-api-access-9d82m\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.021642 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:17.021642 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021625    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.021798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021649    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2c5\" (UniqueName: \"kubernetes.io/projected/ebc19999-3d20-4929-93a6-cfa85a9074d3-kube-api-access-ln2c5\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.021798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021707    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-ovn\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.021798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-bin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.021798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-systemd\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.021798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021770    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-sys\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.021798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021784    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-config\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovn-node-metrics-cert\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fmf\" (UniqueName: \"kubernetes.io/projected/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-kube-api-access-r6fmf\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-var-lib-kubelet\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021884    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-agent-certs\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021937    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-conf\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021953    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-systemd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.021992    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x95b\" (UniqueName: \"kubernetes.io/projected/acca8ca1-d447-4a1f-bd99-8dc48014e033-kube-api-access-5x95b\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.022019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022006    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-modprobe-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022026    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysconfig\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-kubernetes\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022054    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-run\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022126    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-serviceca\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-cnibin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022252    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-netns\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-slash\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.022354 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-konnectivity-ca\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022382    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-system-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022415    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-host\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-multus\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022464    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-hostroot\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-etc-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022521    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acca8ca1-d447-4a1f-bd99-8dc48014e033-iptables-alerter-script\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022544    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-dbus\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-kubelet\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-daemon-config\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022621    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-etc-kubernetes\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022658    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-conf-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc19999-3d20-4929-93a6-cfa85a9074d3-tmp-dir\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-node-log\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.022779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022760    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-bin\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-cni-binary-copy\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-multus-certs\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-host\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-device-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-netns\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022907    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58pb\" (UniqueName: \"kubernetes.io/projected/b5e89546-8707-4c00-9bf2-826c6fbcd761-kube-api-access-k58pb\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022929    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022950    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95vb9\" (UniqueName: \"kubernetes.io/projected/a56de33d-a1ed-44e6-863f-11ae45cec41f-kube-api-access-95vb9\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.022994    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebc19999-3d20-4929-93a6-cfa85a9074d3-hosts-file\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023016    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-netd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-script-lib\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023084    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-systemd-units\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023099    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acca8ca1-d447-4a1f-bd99-8dc48014e033-host-slash\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.023209 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023120    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-kubelet-config\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.023705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-k8s-cni-cncf-io\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.023705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023166    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-lib-modules\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.023705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023225    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.023705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023250    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.023705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023286    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-log-socket\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.023705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.023311    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-kubelet\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.046488 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.046471    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:17.046583 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.046495    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:17.046583 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.046550    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6xhn\""
Oct 30 18:36:17.047233 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.047218    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:17.047295 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.047234    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:17.047352 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.047293    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7txc\""
Oct 30 18:36:17.057716 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.057693    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:17.057921 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.057907    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:17.057984 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.057958    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6mkd\""
Oct 30 18:36:17.058116 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058099    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:17.058116 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058107    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-djndp\""
Oct 30 18:36:17.058223 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058201    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:17.058263 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058247    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:17.058303 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058275    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:17.058303 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058284    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sslwz\""
Oct 30 18:36:17.058397 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058347    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:17.058397 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058374    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:17.058397 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058392    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:17.058498 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.058458    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:17.059041 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.059026    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:17.059119 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.059043    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:17.059359 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.059347    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:17.093017 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.092967    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"82007190ff1dd353d8af659976a136ef","Type":"ContainerStarted","Data":"0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32"}
Oct 30 18:36:17.093999 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.093980    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"7be7e0652d2c7a6c3979448764fb16b6","Type":"ContainerStarted","Data":"7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797"}
Oct 30 18:36:17.097271 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.097253    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 18:36:17.097366 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.097296    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-khfg8"
Oct 30 18:36:17.109771 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.109753    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:17.123928 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.123903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-conf\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124013 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.123936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.124013 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.123958    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-systemd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124013 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.123982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x95b\" (UniqueName: \"kubernetes.io/projected/acca8ca1-d447-4a1f-bd99-8dc48014e033-kube-api-access-5x95b\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124005    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-modprobe-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124025    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-systemd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124043    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysconfig\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124048    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124067    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-conf\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124088    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysconfig\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-kubernetes\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124130    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124142 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124141    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-modprobe-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-run\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124199    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-serviceca\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-cnibin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124247    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-kubernetes\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124378    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-netns\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-slash\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124453    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-netns\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124486 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-konnectivity-ca\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124492    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-system-cni-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124470    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-slash\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124534    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124573    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-cnibin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124546    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-run\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-system-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-host\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124675    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-system-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-multus\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124711    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-host\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124221    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124738    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-hostroot\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124738    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-multus\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-hostroot\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-etc-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-etc-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.124926 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124807    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acca8ca1-d447-4a1f-bd99-8dc48014e033-iptables-alerter-script\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124862    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-dbus\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124958    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-kubelet\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124976    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-daemon-config\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.124990    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-etc-kubernetes\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125014    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125027    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-dbus\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125043    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n59hl\" (UniqueName: \"kubernetes.io/projected/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-kube-api-access-n59hl\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125054    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-kubelet\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-conf-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125077    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-etc-kubernetes\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125101    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc19999-3d20-4929-93a6-cfa85a9074d3-tmp-dir\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-node-log\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125132    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-conf-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-bin\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125159    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-node-log\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-cni-binary-copy\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.125696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125184    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-bin\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-multus-certs\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125225    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-host\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-device-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125261    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-multus-certs\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125265    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-netns\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125292    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-netns\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125309    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58pb\" (UniqueName: \"kubernetes.io/projected/b5e89546-8707-4c00-9bf2-826c6fbcd761-kube-api-access-k58pb\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125337    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-serviceca\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125348    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-host\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vb9\" (UniqueName: \"kubernetes.io/projected/a56de33d-a1ed-44e6-863f-11ae45cec41f-kube-api-access-95vb9\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125416    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125428    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc19999-3d20-4929-93a6-cfa85a9074d3-tmp-dir\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebc19999-3d20-4929-93a6-cfa85a9074d3-hosts-file\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125388    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-device-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125451    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acca8ca1-d447-4a1f-bd99-8dc48014e033-iptables-alerter-script\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125456    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.126392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125524    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebc19999-3d20-4929-93a6-cfa85a9074d3-hosts-file\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125524    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-netd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125594    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-script-lib\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125613    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-netd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cnibin\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-systemd-units\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125668    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acca8ca1-d447-4a1f-bd99-8dc48014e033-host-slash\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125692    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-systemd-units\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125699    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-kubelet-config\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125726    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-kubelet-config\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125727    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acca8ca1-d447-4a1f-bd99-8dc48014e033-host-slash\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125737    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-k8s-cni-cncf-io\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125766    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-k8s-cni-cncf-io\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-lib-modules\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.127124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125905    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125912    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-log-socket\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125924    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-log-socket\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125949    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-kubelet\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125976    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125952    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-lib-modules\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.125989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.126062    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126075    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-script-lib\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126090    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-kubelet\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-tmp\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.126133    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret podName:daacb5ca-b3fa-4344-add9-c451fc1e5e25 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.626118202 +0000 UTC m=+3.122010424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret") pod "global-pull-secret-syncer-k8tk8" (UID: "daacb5ca-b3fa-4344-add9-c451fc1e5e25") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-var-lib-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.127882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-env-overrides\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126228    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126246    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-var-lib-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126253    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-socket-dir-parent\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126277    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-socket-dir-parent\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126281    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-tuned\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgrz\" (UniqueName: \"kubernetes.io/projected/19d1dc0f-27db-4784-916c-dbf422a4d602-kube-api-access-ddgrz\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d82m\" (UniqueName: \"kubernetes.io/projected/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kube-api-access-9d82m\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2s4p\" (UniqueName: \"kubernetes.io/projected/e1321427-04cc-401b-8df9-43b3664b0720-kube-api-access-z2s4p\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126735    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126720    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126770    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2c5\" (UniqueName: \"kubernetes.io/projected/ebc19999-3d20-4929-93a6-cfa85a9074d3-kube-api-access-ln2c5\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-ovn\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-bin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-systemd\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-sys\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.128597 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-config\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126913    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-env-overrides\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovn-node-metrics-cert\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.126973    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fmf\" (UniqueName: \"kubernetes.io/projected/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-kube-api-access-r6fmf\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.127008    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-var-lib-kubelet\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.127040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-agent-certs\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.127087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-sys\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.127134    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-ovn\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.127173    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-var-lib-kubelet\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.127278    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-bin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.129307 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.127380    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-systemd\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.131064 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.131019    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-tuned\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.131163 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.131144    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovn-node-metrics-cert\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.131310 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.131294    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-tmp\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.146500 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.146483    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:17.146782 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.146765    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:17.146841 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.146786    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:17.146945 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.146933    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vmn22\""
Oct 30 18:36:17.147017 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.146935    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:17.147017 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.146989    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:17.147017 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.147003    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h66ck\""
Oct 30 18:36:17.147017 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.146990    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:17.147159 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.147019    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:17.147159 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.147031    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:17.147159 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.147047    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p85r8\""
Oct 30 18:36:17.147159 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.147049    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thsk7\""
Oct 30 18:36:17.147159 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.147094    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:17.147159 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.147138    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:17.148611 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.148590    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-config\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.149936 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.149918    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-agent-certs\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.155089 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.155067    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-konnectivity-ca\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.156138 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.156121    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-daemon-config\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.156914 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.156899    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-cni-binary-copy\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.184155 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.184129    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:17.184259 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.184159    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:17.184259 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.184167    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58pb\" (UniqueName: \"kubernetes.io/projected/b5e89546-8707-4c00-9bf2-826c6fbcd761-kube-api-access-k58pb\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.184259 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.184172    2574 projected.go:196] Error preparing data for projected volume kube-api-access-l7bmn for pod openshift-network-diagnostics/network-check-target-hktmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:17.184461 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.184269    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn podName:d7639481-1e64-49cc-a292-5cfa0cf76416 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.684250596 +0000 UTC m=+3.180142818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-l7bmn" (UniqueName: "kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn") pod "network-check-target-hktmd" (UID: "d7639481-1e64-49cc-a292-5cfa0cf76416") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:17.184461 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.184366    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x95b\" (UniqueName: \"kubernetes.io/projected/acca8ca1-d447-4a1f-bd99-8dc48014e033-kube-api-access-5x95b\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.185025 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.185002    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vb9\" (UniqueName: \"kubernetes.io/projected/a56de33d-a1ed-44e6-863f-11ae45cec41f-kube-api-access-95vb9\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.185816 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.185792    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fmf\" (UniqueName: \"kubernetes.io/projected/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-kube-api-access-r6fmf\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.186002 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.185987    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgrz\" (UniqueName: \"kubernetes.io/projected/19d1dc0f-27db-4784-916c-dbf422a4d602-kube-api-access-ddgrz\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.186763 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.186750    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2c5\" (UniqueName: \"kubernetes.io/projected/ebc19999-3d20-4929-93a6-cfa85a9074d3-kube-api-access-ln2c5\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.186807 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.186766    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d82m\" (UniqueName: \"kubernetes.io/projected/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kube-api-access-9d82m\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.227921 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.227896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cnibin\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228016 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.227935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228016 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.227953    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s4p\" (UniqueName: \"kubernetes.io/projected/e1321427-04cc-401b-8df9-43b3664b0720-kube-api-access-z2s4p\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:17.228016 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.227982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228016 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.228000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-system-cni-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228016 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.227997    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cnibin\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228268 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.228034    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-system-cni-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228268 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.228107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228268 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.228131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228268 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.228149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:17.228268 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.228164    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59hl\" (UniqueName: \"kubernetes.io/projected/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-kube-api-access-n59hl\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.228268 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.228250    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:17.228483 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.228311    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:17.728294158 +0000 UTC m=+3.224186366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:17.229041 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.229018    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.229185 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.229163    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.229235 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.229199    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.229266 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.229236    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.287720 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.287680    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59hl\" (UniqueName: \"kubernetes.io/projected/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-kube-api-access-n59hl\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.288075 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.288057    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s4p\" (UniqueName: \"kubernetes.io/projected/e1321427-04cc-401b-8df9-43b3664b0720-kube-api-access-z2s4p\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:17.302534 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.302516    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:17.308753 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.308738    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:17.311913 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.311896    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19d1dc0f_27db_4784_916c_dbf422a4d602.slice/crio-e37437473b22e164c9f85c1bc66036f5d61c8e2e71bf2ac26504a3e660723845 WatchSource:0}: Error finding container e37437473b22e164c9f85c1bc66036f5d61c8e2e71bf2ac26504a3e660723845: Status 404 returned error can't find the container with id e37437473b22e164c9f85c1bc66036f5d61c8e2e71bf2ac26504a3e660723845
Oct 30 18:36:17.316990 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.316971    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacca8ca1_d447_4a1f_bd99_8dc48014e033.slice/crio-3d79841a1a769d7f8cdd3485314cedb0af23031618e37da1c66bb9d8b2391c2d WatchSource:0}: Error finding container 3d79841a1a769d7f8cdd3485314cedb0af23031618e37da1c66bb9d8b2391c2d: Status 404 returned error can't find the container with id 3d79841a1a769d7f8cdd3485314cedb0af23031618e37da1c66bb9d8b2391c2d
Oct 30 18:36:17.323034 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.323018    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:17.323118 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.323068    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:17.328581 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.328565    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:17.332045 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.332012    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d51c59f_38ab_4311_bf40_cf79edf28d4a.slice/crio-e3c70ee516a89708a242cefbbf143a08d3e24beb903c1b1b14245c0441e1c7f3 WatchSource:0}: Error finding container e3c70ee516a89708a242cefbbf143a08d3e24beb903c1b1b14245c0441e1c7f3: Status 404 returned error can't find the container with id e3c70ee516a89708a242cefbbf143a08d3e24beb903c1b1b14245c0441e1c7f3
Oct 30 18:36:17.335853 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.335830    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc19999_3d20_4929_93a6_cfa85a9074d3.slice/crio-e47c8cbc0af923b5cde2c263466bb43fd9b4f23e12ead943f5c8670666440005 WatchSource:0}: Error finding container e47c8cbc0af923b5cde2c263466bb43fd9b4f23e12ead943f5c8670666440005: Status 404 returned error can't find the container with id e47c8cbc0af923b5cde2c263466bb43fd9b4f23e12ead943f5c8670666440005
Oct 30 18:36:17.337197 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.337179    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c90acf4_d9a8_4478_a97d_cc5bfe500fa0.slice/crio-6e0626d39502c3fc05873906b92be1a6ef36223010e13cdfa69152ae3e504482 WatchSource:0}: Error finding container 6e0626d39502c3fc05873906b92be1a6ef36223010e13cdfa69152ae3e504482: Status 404 returned error can't find the container with id 6e0626d39502c3fc05873906b92be1a6ef36223010e13cdfa69152ae3e504482
Oct 30 18:36:17.343619 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.343595    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2kfph"
Oct 30 18:36:17.343619 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.343609    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:17.346748 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.346727    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:17.352033 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.352014    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56de33d_a1ed_44e6_863f_11ae45cec41f.slice/crio-e63e724d48f0f19b7214cc23c6af655ca1f287b1ee37570f91201c4145de257d WatchSource:0}: Error finding container e63e724d48f0f19b7214cc23c6af655ca1f287b1ee37570f91201c4145de257d: Status 404 returned error can't find the container with id e63e724d48f0f19b7214cc23c6af655ca1f287b1ee37570f91201c4145de257d
Oct 30 18:36:17.352527 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.352506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:17.355510 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.355489    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod511cf18c_8d23_40e7_9e68_f683e3fe9e7d.slice/crio-8e0ff618735547c8ea5cf78e9c01829e04b0d9871061958c101fd654355b8203 WatchSource:0}: Error finding container 8e0ff618735547c8ea5cf78e9c01829e04b0d9871061958c101fd654355b8203: Status 404 returned error can't find the container with id 8e0ff618735547c8ea5cf78e9c01829e04b0d9871061958c101fd654355b8203
Oct 30 18:36:17.360784 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.360765    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e89546_8707_4c00_9bf2_826c6fbcd761.slice/crio-e292958ee020bc49993fbd96dc254942c8ac03afeaf637d1c24b6989d9d6582d WatchSource:0}: Error finding container e292958ee020bc49993fbd96dc254942c8ac03afeaf637d1c24b6989d9d6582d: Status 404 returned error can't find the container with id e292958ee020bc49993fbd96dc254942c8ac03afeaf637d1c24b6989d9d6582d
Oct 30 18:36:17.367000 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.366985    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:17.375847 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:17.375831    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77bb5229_a381_4bf0_82ee_e4eb959cf7eb.slice/crio-1ac122194e6e5df57c31b21a993a4ec0409a7bbafd0c1ffc6143d01e9d00d789 WatchSource:0}: Error finding container 1ac122194e6e5df57c31b21a993a4ec0409a7bbafd0c1ffc6143d01e9d00d789: Status 404 returned error can't find the container with id 1ac122194e6e5df57c31b21a993a4ec0409a7bbafd0c1ffc6143d01e9d00d789
Oct 30 18:36:17.631802 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.631670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:17.631979 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.631840    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:17.631979 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.631915    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret podName:daacb5ca-b3fa-4344-add9-c451fc1e5e25 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:18.631896242 +0000 UTC m=+4.127788460 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret") pod "global-pull-secret-syncer-k8tk8" (UID: "daacb5ca-b3fa-4344-add9-c451fc1e5e25") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:17.732306 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.732275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:17.732465 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:17.732355    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:17.732539 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.732488    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:17.732539 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.732504    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:17.732539 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.732514    2574 projected.go:196] Error preparing data for projected volume kube-api-access-l7bmn for pod openshift-network-diagnostics/network-check-target-hktmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:17.732679 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.732558    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn podName:d7639481-1e64-49cc-a292-5cfa0cf76416 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:18.732543506 +0000 UTC m=+4.228435723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7bmn" (UniqueName: "kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn") pod "network-check-target-hktmd" (UID: "d7639481-1e64-49cc-a292-5cfa0cf76416") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:17.732679 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.732621    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:17.732679 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:17.732654    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:18.732644733 +0000 UTC m=+4.228536951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:18.096601 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.096520    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerStarted","Data":"1ac122194e6e5df57c31b21a993a4ec0409a7bbafd0c1ffc6143d01e9d00d789"}
Oct 30 18:36:18.097930 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.097900    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"e292958ee020bc49993fbd96dc254942c8ac03afeaf637d1c24b6989d9d6582d"}
Oct 30 18:36:18.099124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.099096    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zfktp" event={"ID":"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0","Type":"ContainerStarted","Data":"6e0626d39502c3fc05873906b92be1a6ef36223010e13cdfa69152ae3e504482"}
Oct 30 18:36:18.099897 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.099860    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:17 +0000 UTC" deadline="2028-08-04 10:05:46.422743092 +0000 UTC"
Oct 30 18:36:18.099993 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.099897    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15447h29m28.32285001s"
Oct 30 18:36:18.100264 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.100245    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6975" event={"ID":"ebc19999-3d20-4929-93a6-cfa85a9074d3","Type":"ContainerStarted","Data":"e47c8cbc0af923b5cde2c263466bb43fd9b4f23e12ead943f5c8670666440005"}
Oct 30 18:36:18.101489 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.101465    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5b6w7" event={"ID":"511cf18c-8d23-40e7-9e68-f683e3fe9e7d","Type":"ContainerStarted","Data":"8e0ff618735547c8ea5cf78e9c01829e04b0d9871061958c101fd654355b8203"}
Oct 30 18:36:18.102800 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.102777    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kfph" event={"ID":"a56de33d-a1ed-44e6-863f-11ae45cec41f","Type":"ContainerStarted","Data":"e63e724d48f0f19b7214cc23c6af655ca1f287b1ee37570f91201c4145de257d"}
Oct 30 18:36:18.104092 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.104069    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67" event={"ID":"9d51c59f-38ab-4311-bf40-cf79edf28d4a","Type":"ContainerStarted","Data":"e3c70ee516a89708a242cefbbf143a08d3e24beb903c1b1b14245c0441e1c7f3"}
Oct 30 18:36:18.105415 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.105392    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcsph" event={"ID":"acca8ca1-d447-4a1f-bd99-8dc48014e033","Type":"ContainerStarted","Data":"3d79841a1a769d7f8cdd3485314cedb0af23031618e37da1c66bb9d8b2391c2d"}
Oct 30 18:36:18.106578 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.106554    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8v44" event={"ID":"19d1dc0f-27db-4784-916c-dbf422a4d602","Type":"ContainerStarted","Data":"e37437473b22e164c9f85c1bc66036f5d61c8e2e71bf2ac26504a3e660723845"}
Oct 30 18:36:18.639659 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.639619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:18.639851 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.639796    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:18.639915 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.639859    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret podName:daacb5ca-b3fa-4344-add9-c451fc1e5e25 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:20.639842609 +0000 UTC m=+6.135734827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret") pod "global-pull-secret-syncer-k8tk8" (UID: "daacb5ca-b3fa-4344-add9-c451fc1e5e25") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:18.740830 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.740798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:18.740999 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:18.740864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:18.740999 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.740987    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:18.741104 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.741047    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:20.741029107 +0000 UTC m=+6.236921314 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:18.741163 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.741131    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:18.741163 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.741148    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:18.741163 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.741161    2574 projected.go:196] Error preparing data for projected volume kube-api-access-l7bmn for pod openshift-network-diagnostics/network-check-target-hktmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:18.741376 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:18.741197    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn podName:d7639481-1e64-49cc-a292-5cfa0cf76416 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:20.741185898 +0000 UTC m=+6.237078109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7bmn" (UniqueName: "kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn") pod "network-check-target-hktmd" (UID: "d7639481-1e64-49cc-a292-5cfa0cf76416") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:19.098072 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:19.098043    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:19.098674 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:19.098175    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:19.098674 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:19.098261    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:19.098674 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:19.098348    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:19.098674 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:19.098419    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:19.098674 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:19.098500    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:19.100195 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:19.100162    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:17 +0000 UTC" deadline="2028-08-09 17:54:19.873101416 +0000 UTC"
Oct 30 18:36:19.100195 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:19.100189    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15575h18m0.772916401s"
Oct 30 18:36:20.119459 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:20.118659    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"82007190ff1dd353d8af659976a136ef","Type":"ContainerStarted","Data":"1dbef6ba6555bead4a8dc3aa8e0de1ef45fc9a0e2f92292e54b754cef3940b03"}
Oct 30 18:36:20.238804 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:20.238498    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal" podStartSLOduration=4.238477008 podStartE2EDuration="4.238477008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:20.238207267 +0000 UTC m=+5.734099494" watchObservedRunningTime="2026-10-30 18:36:20.238477008 +0000 UTC m=+5.734369233"
Oct 30 18:36:20.659105 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:20.659047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:20.659279 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.659229    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:20.659378 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.659314    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret podName:daacb5ca-b3fa-4344-add9-c451fc1e5e25 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:24.659295563 +0000 UTC m=+10.155187780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret") pod "global-pull-secret-syncer-k8tk8" (UID: "daacb5ca-b3fa-4344-add9-c451fc1e5e25") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:20.759806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:20.759879    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.760042    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.760065    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.760079    2574 projected.go:196] Error preparing data for projected volume kube-api-access-l7bmn for pod openshift-network-diagnostics/network-check-target-hktmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.760142    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn podName:d7639481-1e64-49cc-a292-5cfa0cf76416 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:24.760123167 +0000 UTC m=+10.256015388 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7bmn" (UniqueName: "kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn") pod "network-check-target-hktmd" (UID: "d7639481-1e64-49cc-a292-5cfa0cf76416") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.760227    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:20.760292 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:20.760270    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:24.760258272 +0000 UTC m=+10.256150492 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:21.089503 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:21.089469    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:21.089695 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:21.089609    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:21.089695 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:21.089676    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:21.089847 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:21.089745    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:21.089847 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:21.089802    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:21.089952 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:21.089864    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:21.124854 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:21.124821    2574 generic.go:356] "Generic (PLEG): container finished" podID="7be7e0652d2c7a6c3979448764fb16b6" containerID="158398e0acf3fb176ba22f7d6b93572660a1088c1c59ff77803d35f251381ce3" exitCode=0
Oct 30 18:36:21.125207 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:21.124909    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"7be7e0652d2c7a6c3979448764fb16b6","Type":"ContainerDied","Data":"158398e0acf3fb176ba22f7d6b93572660a1088c1c59ff77803d35f251381ce3"}
Oct 30 18:36:21.134397 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:21.134369    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerStarted","Data":"da595da6965f804b3bb01ac9e2ebaf716e55737f8dd8cac12fcb41ea636469ff"}
Oct 30 18:36:22.144472 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:22.144302    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5b6w7" event={"ID":"511cf18c-8d23-40e7-9e68-f683e3fe9e7d","Type":"ContainerStarted","Data":"63f530249fde3aa3e2b95f5f889c973ef7bf2ff653f942bcf377941f0b62dbb7"}
Oct 30 18:36:22.152305 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:22.151892    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"7be7e0652d2c7a6c3979448764fb16b6","Type":"ContainerStarted","Data":"60bdc0bd3cec7891bd1e944d2d86d7253931c4e94d9a1c126fdda7f45bafb9c2"}
Oct 30 18:36:22.156541 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:22.155901    2574 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="da595da6965f804b3bb01ac9e2ebaf716e55737f8dd8cac12fcb41ea636469ff" exitCode=0
Oct 30 18:36:22.156541 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:22.155944    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerDied","Data":"da595da6965f804b3bb01ac9e2ebaf716e55737f8dd8cac12fcb41ea636469ff"}
Oct 30 18:36:22.381770 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:22.381717    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5b6w7" podStartSLOduration=2.921826317 podStartE2EDuration="7.381698742s" totalImagesPullingTime="4.459872425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:22.263764279 +0000 UTC m=+7.759656536" watchObservedRunningTime="2026-10-30 18:36:22.381698742 +0000 UTC m=+7.877590968"
Oct 30 18:36:22.381971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:22.381939    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal" podStartSLOduration=6.3819323 podStartE2EDuration="6.3819323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:22.381262652 +0000 UTC m=+7.877154879" watchObservedRunningTime="2026-10-30 18:36:22.3819323 +0000 UTC m=+7.877824526"
Oct 30 18:36:23.094376 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:23.094349    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:23.094551 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:23.094484    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:23.095078 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:23.094909    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:23.095078 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:23.095005    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:23.095078 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:23.095053    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:23.095276 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:23.095117    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:23.162492 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:23.162458    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zfktp" event={"ID":"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0","Type":"ContainerStarted","Data":"b254c7e4765d13430d5169530d6ffebd562fe951fba7db1cf512633bbe8bc692"}
Oct 30 18:36:24.168048 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:24.168011    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67" event={"ID":"9d51c59f-38ab-4311-bf40-cf79edf28d4a","Type":"ContainerStarted","Data":"82107e04843e185dc3037f6b745a6efe999d94277ef8fdc35d6f889162f582f7"}
Oct 30 18:36:24.693398 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:24.693294    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:24.693574 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.693479    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:24.693574 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.693545    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret podName:daacb5ca-b3fa-4344-add9-c451fc1e5e25 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:32.693526503 +0000 UTC m=+18.189418721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret") pod "global-pull-secret-syncer-k8tk8" (UID: "daacb5ca-b3fa-4344-add9-c451fc1e5e25") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:24.794530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:24.794601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.794720    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.794743    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.794777    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:32.794759945 +0000 UTC m=+18.290652150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.794773    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.794819    2574 projected.go:196] Error preparing data for projected volume kube-api-access-l7bmn for pod openshift-network-diagnostics/network-check-target-hktmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:24.795091 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:24.794872    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn podName:d7639481-1e64-49cc-a292-5cfa0cf76416 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:32.794856544 +0000 UTC m=+18.290748749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7bmn" (UniqueName: "kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn") pod "network-check-target-hktmd" (UID: "d7639481-1e64-49cc-a292-5cfa0cf76416") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:25.094717 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.093780    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:25.094717 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:25.093910    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:25.094717 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.094368    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:25.094717 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:25.094464    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:25.094717 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.094537    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:25.094717 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:25.094623    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:25.567675 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.567555    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:25.715119 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.715083    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:25.983913 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.983805    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:25.715095744Z","UUID":"9dabde29-57fb-484b-95e1-edd042d0643e","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:25.985986 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.985965    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:25.986100 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:25.985994    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:26.173736 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.173659    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s6975" event={"ID":"ebc19999-3d20-4929-93a6-cfa85a9074d3","Type":"ContainerStarted","Data":"c887f62b3a7f53a1ea5cf8ffcf91982d403520800ac9fed3aa94afe4ebc2c76f"}
Oct 30 18:36:26.176014 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.175982    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2kfph" event={"ID":"a56de33d-a1ed-44e6-863f-11ae45cec41f","Type":"ContainerStarted","Data":"30bf1e745512d73cc0ebc87e986aa7e46badc62e7ce9c8397f553689fd7dcd72"}
Oct 30 18:36:26.179453 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.179430    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67" event={"ID":"9d51c59f-38ab-4311-bf40-cf79edf28d4a","Type":"ContainerStarted","Data":"2892ed70c1e9b142c689f4acdbebb3ae52cc43a2773277c172810fba1bcc7708"}
Oct 30 18:36:26.291283 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.291227    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-s6975" podStartSLOduration=2.891956483 podStartE2EDuration="11.291213476s" totalImagesPullingTime="8.399256993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:26.291097412 +0000 UTC m=+11.786989639" watchObservedRunningTime="2026-10-30 18:36:26.291213476 +0000 UTC m=+11.787105701"
Oct 30 18:36:26.291465 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.291346    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zfktp" podStartSLOduration=6.272234696 podStartE2EDuration="11.291339384s" totalImagesPullingTime="5.019104688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:23.280610537 +0000 UTC m=+8.776502766" watchObservedRunningTime="2026-10-30 18:36:26.291339384 +0000 UTC m=+11.787231605"
Oct 30 18:36:26.348626 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.348350    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:26.349048 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.349020    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:26.410479 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:26.410413    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2kfph" podStartSLOduration=3.172492413 podStartE2EDuration="11.410394563s" totalImagesPullingTime="8.23790215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:26.410228749 +0000 UTC m=+11.906120976" watchObservedRunningTime="2026-10-30 18:36:26.410394563 +0000 UTC m=+11.906286793"
Oct 30 18:36:27.090917 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:27.090884    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:27.091407 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:27.091019    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:27.091407 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:27.090901    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:27.091407 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:27.091095    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:27.091407 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:27.090894    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:27.091407 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:27.091170    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:27.185308 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:27.185273    2574 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="29877a2ab78d5c27f3374e88435da416203f51d6c97f8cc3a73a7a45a6682427" exitCode=0
Oct 30 18:36:27.185495 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:27.185412    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerDied","Data":"29877a2ab78d5c27f3374e88435da416203f51d6c97f8cc3a73a7a45a6682427"}
Oct 30 18:36:27.186679 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:27.186541    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:28.190271 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:28.190231    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67" event={"ID":"9d51c59f-38ab-4311-bf40-cf79edf28d4a","Type":"ContainerStarted","Data":"4d4796ae8799f64b637ba4e4f2d0541a1a2d3e0486dd28535ae7d23c9a68716e"}
Oct 30 18:36:28.196141 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:28.196103    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fcsph" event={"ID":"acca8ca1-d447-4a1f-bd99-8dc48014e033","Type":"ContainerStarted","Data":"8ff50f080817e59b47ba3f94d8d4f104c572b4c3900df7b5f7dbc5a7fb7b953e"}
Oct 30 18:36:28.197676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:28.197635    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-m8v44" event={"ID":"19d1dc0f-27db-4784-916c-dbf422a4d602","Type":"ContainerStarted","Data":"999a3c9e9754fb7d2e2dd7940c4b93c7ddf2c6a339d3c5c4a5d7a12eda6233fd"}
Oct 30 18:36:28.309567 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:28.309472    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67" podStartSLOduration=3.486799229 podStartE2EDuration="13.309459855s" totalImagesPullingTime="9.822660626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:28.309015374 +0000 UTC m=+13.804907601" watchObservedRunningTime="2026-10-30 18:36:28.309459855 +0000 UTC m=+13.805352078"
Oct 30 18:36:28.425556 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:28.425494    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fcsph" podStartSLOduration=5.098010799 podStartE2EDuration="13.425479562s" totalImagesPullingTime="8.327468763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:28.42538613 +0000 UTC m=+13.921278357" watchObservedRunningTime="2026-10-30 18:36:28.425479562 +0000 UTC m=+13.921371789"
Oct 30 18:36:28.546559 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:28.546507    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-m8v44" podStartSLOduration=2.896792445 podStartE2EDuration="13.546490794s" totalImagesPullingTime="10.649698349s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:28.546119866 +0000 UTC m=+14.042012093" watchObservedRunningTime="2026-10-30 18:36:28.546490794 +0000 UTC m=+14.042383020"
Oct 30 18:36:29.089744 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:29.089650    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:29.089744 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:29.089699    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:29.089744 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:29.089704    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:29.089960 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:29.089800    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:29.089960 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:29.089876    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:29.090018 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:29.089956    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:29.201800 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:29.201769    2574 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="2dbfce9b0f76a54fcae83e59fe5af47eb83b0c6fc5c58b734554f55ad2188ef2" exitCode=0
Oct 30 18:36:29.202365 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:29.201866    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerDied","Data":"2dbfce9b0f76a54fcae83e59fe5af47eb83b0c6fc5c58b734554f55ad2188ef2"}
Oct 30 18:36:31.089958 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:31.089703    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:31.090549 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:31.089703    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:31.090549 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:31.090075    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:31.090549 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:31.089703    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:31.090549 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:31.090152    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:31.090549 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:31.090231    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:31.208876 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:31.208839    2574 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="218db7ddac86120c26c1a3300cefa56dc37aa32f17217ca7e6bc663f3590e567" exitCode=0
Oct 30 18:36:31.209028 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:31.208901    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerDied","Data":"218db7ddac86120c26c1a3300cefa56dc37aa32f17217ca7e6bc663f3590e567"}
Oct 30 18:36:32.752130 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:32.752100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:32.752462 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.752257    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:32.752462 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.752299    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret podName:daacb5ca-b3fa-4344-add9-c451fc1e5e25 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.752287756 +0000 UTC m=+34.248179961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret") pod "global-pull-secret-syncer-k8tk8" (UID: "daacb5ca-b3fa-4344-add9-c451fc1e5e25") : object "kube-system"/"original-pull-secret" not registered
Oct 30 18:36:32.853270 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:32.853238    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:32.853391 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:32.853306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:32.853474 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.853457    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:32.853524 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.853519    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.853499779 +0000 UTC m=+34.349391984 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 18:36:32.853618 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.853603    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 18:36:32.853681 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.853622    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 18:36:32.853681 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.853634    2574 projected.go:196] Error preparing data for projected volume kube-api-access-l7bmn for pod openshift-network-diagnostics/network-check-target-hktmd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:32.853681 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:32.853670    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn podName:d7639481-1e64-49cc-a292-5cfa0cf76416 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.853657031 +0000 UTC m=+34.349549235 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-l7bmn" (UniqueName: "kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn") pod "network-check-target-hktmd" (UID: "d7639481-1e64-49cc-a292-5cfa0cf76416") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 18:36:33.090613 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.090392    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:33.090613 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.090402    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:33.090613 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.090547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:33.090613 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:33.090523    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:33.090909 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:33.090634    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:33.090909 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:33.090700    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:33.215895 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.215860    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"5c265e595e0d86007f8a7a870a69d5366d8bbf03677c6c0346f33b2bd7e2fd2d"}
Oct 30 18:36:33.216083 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.215943    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"f48ce1c72ebabcf931413b749ef3f976427c3250f395ace42aa73aece0f7a01f"}
Oct 30 18:36:33.216083 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.215985    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"e6e80ea079f476bcfd7719026e28b8336adc805cdda81265b35f5f31d34248b7"}
Oct 30 18:36:33.216083 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.216028    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"f9839f8520cb9ae09c41b11a740ba1a68e07f5c52357cf18632b1856fb3d436d"}
Oct 30 18:36:33.216083 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.216069    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"4dca6dded6e5b7b1479a7e74ee884c9a210a9e209b4b0665cf15b7e0c4e07bc3"}
Oct 30 18:36:33.216240 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:33.216110    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"77026a0d18ecd35111eee36c1ec1ddb040fd43135b6e809dcb80807ec771977b"}
Oct 30 18:36:34.220959 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:34.220924    2574 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="e30d10b2a7559143fae6fbf6de48715885b63931b69c7d1a5fe9903e6e2259c6" exitCode=0
Oct 30 18:36:34.221428 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:34.220988    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerDied","Data":"e30d10b2a7559143fae6fbf6de48715885b63931b69c7d1a5fe9903e6e2259c6"}
Oct 30 18:36:35.091454 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:35.091421    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:35.091594 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:35.091502    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:35.091594 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:35.091542    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:35.091594 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:35.091533    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:35.091717 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:35.091633    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:35.091757 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:35.091727    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:35.225848 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:35.225773    2574 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="7a38b0370b47fc210609aa7b088de80e74045c4afed00f71a555101464b434c0" exitCode=0
Oct 30 18:36:35.225848 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:35.225845    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerDied","Data":"7a38b0370b47fc210609aa7b088de80e74045c4afed00f71a555101464b434c0"}
Oct 30 18:36:35.229198 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:35.229179    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"ecfc1f137468c36418dc7a012654c05b09dd8370f4811cbe8d43b833112acf40"}
Oct 30 18:36:36.234581 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:36.234540    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" event={"ID":"77bb5229-a381-4bf0-82ee-e4eb959cf7eb","Type":"ContainerStarted","Data":"8bc22de5851d69111ebb5059be3d987e226014803d094069d810195ce0ae1ef3"}
Oct 30 18:36:36.356591 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:36.356545    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-m4dhb" podStartSLOduration=8.757814355 podStartE2EDuration="21.356533069s" totalImagesPullingTime="12.598718714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:36.356272173 +0000 UTC m=+21.852164400" watchObservedRunningTime="2026-10-30 18:36:36.356533069 +0000 UTC m=+21.852425294"
Oct 30 18:36:37.090124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.090091    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:37.090274 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.090100    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:37.090274 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.090209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:37.090274 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:37.090222    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:37.090443 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:37.090342    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:37.090443 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:37.090421    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:37.239857 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.239740    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" event={"ID":"b5e89546-8707-4c00-9bf2-826c6fbcd761","Type":"ContainerStarted","Data":"817cd482ab37abe1930363a01f076ac7e5604a1356eeb86499a4405343b750cb"}
Oct 30 18:36:37.240516 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.239962    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:37.240516 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.240038    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:37.256185 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.256158    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:37.256292 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.256252    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:37.497864 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:37.497771    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89" podStartSLOduration=7.359563971 podStartE2EDuration="22.497757788s" totalImagesPullingTime="15.138193817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:37.4462473 +0000 UTC m=+22.942139539" watchObservedRunningTime="2026-10-30 18:36:37.497757788 +0000 UTC m=+22.993650013"
Oct 30 18:36:38.089576 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:38.089552    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:39.089901 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.089864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:39.090458 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.089917    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:39.090458 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.089930    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:39.090458 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:39.089986    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:39.090458 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:39.090057    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:39.090458 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:39.090117    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:39.603859 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.603827    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k8tk8"]
Oct 30 18:36:39.604019 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.603919    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:39.604019 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:39.603997    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:39.606930 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.606912    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hktmd"]
Oct 30 18:36:39.607008 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.606985    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:39.607123 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:39.607058    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:39.607500 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.607480    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fv5vj"]
Oct 30 18:36:39.607581 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:39.607568    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:39.607691 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:39.607668    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:41.089376 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:41.089348    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:41.089737 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:41.089351    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:41.089737 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:41.089466    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:41.089737 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:41.089467    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:41.089737 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:41.089615    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:41.089737 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:41.089699    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:43.089663 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:43.089627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:43.089663 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:43.089654    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:43.090183 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:43.089638    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:43.090183 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:43.089773    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:43.090183 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:43.089876    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:43.090183 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:43.089939    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:45.090034 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:45.090009    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:45.090359 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:45.090091    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k8tk8" podUID="daacb5ca-b3fa-4344-add9-c451fc1e5e25"
Oct 30 18:36:45.090359 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:45.090106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:45.090359 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:45.090228    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fv5vj" podUID="e1321427-04cc-401b-8df9-43b3664b0720"
Oct 30 18:36:45.090359 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:45.090274    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:45.090494 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:45.090366    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hktmd" podUID="d7639481-1e64-49cc-a292-5cfa0cf76416"
Oct 30 18:36:46.766750 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.766721    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeReady"
Oct 30 18:36:46.767143 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.766833    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 18:36:46.885060 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.885028    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vvw8v"]
Oct 30 18:36:46.888067 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.888048    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cw5s"]
Oct 30 18:36:46.888192 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.888170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vvw8v"
Oct 30 18:36:46.890836 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.890816    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-b895475ff-2t4mb"]
Oct 30 18:36:46.890954 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.890936    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8cw5s"
Oct 30 18:36:46.893543 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.893525    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rw5pb"]
Oct 30 18:36:46.893622 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.893611    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:46.896946 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.896928    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-s4bss"]
Oct 30 18:36:46.897061 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.897045    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rw5pb"
Oct 30 18:36:46.899438 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.899424    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"]
Oct 30 18:36:46.899534 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.899520    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:46.902028 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.902009    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b457d476c-cmkxg"]
Oct 30 18:36:46.902124 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.902108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:46.904842 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.904826    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sfn74"]
Oct 30 18:36:46.904936 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.904923    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:46.907610 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.907591    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gbsgk"]
Oct 30 18:36:46.907708 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.907693    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:46.910443 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.910427    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"]
Oct 30 18:36:46.910522 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.910511    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:46.913403 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.913386    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"]
Oct 30 18:36:46.913513 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.913497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:46.916591 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.916572    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"]
Oct 30 18:36:46.916676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.916663    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"
Oct 30 18:36:46.919428 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.919410    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mx59k"]
Oct 30 18:36:46.919546 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.919522    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:46.922499 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.922478    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"]
Oct 30 18:36:46.922707 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.922691    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:46.925547 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.925530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:46.940863 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.940840    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8d9cdb74b-n2d7w"]
Oct 30 18:36:46.946179 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.946157    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dhgxq"]
Oct 30 18:36:46.946278 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.946255    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:46.948760 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.948737    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:36:46.948905 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.948892    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:36:46.949546 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.949530    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:36:46.949661 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.949572    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:36:46.949661 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.949614    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7jwft\""
Oct 30 18:36:46.949763 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.949747    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"]
Oct 30 18:36:46.949856 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.949843    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:46.954764 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.954746    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vvw8v"]
Oct 30 18:36:46.954841 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.954785    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cw5s"]
Oct 30 18:36:46.954841 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.954814    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"]
Oct 30 18:36:46.954841 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.954834    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-s4bss"]
Oct 30 18:36:46.954978 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.954855    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rw5pb"]
Oct 30 18:36:46.954978 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.954872    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b895475ff-2t4mb"]
Oct 30 18:36:46.954978 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.954929    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"
Oct 30 18:36:46.957638 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.957619    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7pwv\" (UniqueName: \"kubernetes.io/projected/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb-kube-api-access-m7pwv\") pod \"kas-connection-checker-685895c7c9-vvw8v\" (UID: \"a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-vvw8v"
Oct 30 18:36:46.988544 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:46.988521    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sfn74"]
Oct 30 18:36:47.016646 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.016627    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:36:47.017683 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.017630    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbsgk"]
Oct 30 18:36:47.018413 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.018398    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"]
Oct 30 18:36:47.019077 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.019052    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"]
Oct 30 18:36:47.019842 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.019825    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d9cdb74b-n2d7w"]
Oct 30 18:36:47.020500 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.020483    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b457d476c-cmkxg"]
Oct 30 18:36:47.021052 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.021039    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"]
Oct 30 18:36:47.021661 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.021645    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mx59k"]
Oct 30 18:36:47.022789 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.022773    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"]
Oct 30 18:36:47.029413 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.029396    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhgxq"]
Oct 30 18:36:47.032313 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.032297    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"]
Oct 30 18:36:47.051423 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.051397    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:36:47.051497 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.051430    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.051676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.051660    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.052015 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052004    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.052392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052368    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:36:47.052471 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052449    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:36:47.052522 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052476    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:36:47.052567 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052498    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.052567 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052541    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.052767 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052749    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.052880 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052864    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.052977 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.052903    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.053239 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053208    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:47.053647 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053356    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.053647 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053468    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.053647 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053524    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.053647 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053638    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:36:47.053900 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053678    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.053992 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053975    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.054050 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.053976    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.054101 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054070    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.054278 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054261    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:36:47.054509 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054491    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.054666 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054650    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.054753 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054654    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.054852 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054837    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:47.054930 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054777    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:47.054981 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.054943    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:47.057923 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.057896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e96c52d-079e-4b90-b722-b69ad8b12730-serving-cert\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.058008 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.057939    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.058061 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvfsh\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.058117 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058091    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.058167 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058123    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b240bd-b789-4575-8a22-f2773e4ef78d-cert\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:47.058167 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46xgz\" (UniqueName: \"kubernetes.io/projected/1e6a72c5-3683-4fc8-8525-e82baed6a4ad-kube-api-access-46xgz\") pod \"kas-connection-checker-685895c7c9-rw5pb\" (UID: \"1e6a72c5-3683-4fc8-8525-e82baed6a4ad\") " pod="kube-system/kas-connection-checker-685895c7c9-rw5pb"
Oct 30 18:36:47.058290 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058268    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-trusted-ca\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.058359 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058304    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.058359 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058346    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-snapshots\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.058428 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljjhb\" (UniqueName: \"kubernetes.io/projected/ed959a30-6e00-49b5-aceb-c543a68b9d3e-kube-api-access-ljjhb\") pod \"volume-data-source-validator-58c9f5f46d-rcf9k\" (UID: \"ed959a30-6e00-49b5-aceb-c543a68b9d3e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"
Oct 30 18:36:47.058465 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.058502 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058481    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7ttf\" (UniqueName: \"kubernetes.io/projected/8e96c52d-079e-4b90-b722-b69ad8b12730-kube-api-access-k7ttf\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.058552 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058531    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.058604 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.058657 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.058691 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058658    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjnpk\" (UniqueName: \"kubernetes.io/projected/4a159864-3010-4195-84de-1407b16eb5c2-kube-api-access-pjnpk\") pod \"kas-connection-checker-685895c7c9-8cw5s\" (UID: \"4a159864-3010-4195-84de-1407b16eb5c2\") " pod="kube-system/kas-connection-checker-685895c7c9-8cw5s"
Oct 30 18:36:47.058691 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.058788 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.058788 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-stats-auth\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.058788 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.058971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.058971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058828    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.058971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9790e2f-d93d-47f4-aa83-6c8a54567513-serving-cert\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.058971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058916    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pwv\" (UniqueName: \"kubernetes.io/projected/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb-kube-api-access-m7pwv\") pod \"kas-connection-checker-685895c7c9-vvw8v\" (UID: \"a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-vvw8v"
Oct 30 18:36:47.058971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f92791-c615-4d6e-88e8-1a3f275aa32a-serving-cert\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.058971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058955    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.058971 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.058973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9fdf\" (UniqueName: \"kubernetes.io/projected/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-kube-api-access-w9fdf\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.059222 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhrlf\" (UniqueName: \"kubernetes.io/projected/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-kube-api-access-mhrlf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.059222 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059094    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.059222 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059110    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.059222 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059139    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f92791-c615-4d6e-88e8-1a3f275aa32a-config\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.059222 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-tmp\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.059403 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059230    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.059403 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059290    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-default-certificate\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.059403 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.059403 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-config\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.059514 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059412    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8jrs\" (UniqueName: \"kubernetes.io/projected/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-kube-api-access-c8jrs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.059514 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhmtd\" (UniqueName: \"kubernetes.io/projected/c6f92791-c615-4d6e-88e8-1a3f275aa32a-kube-api-access-hhmtd\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.059514 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mprt\" (UniqueName: \"kubernetes.io/projected/80b240bd-b789-4575-8a22-f2773e4ef78d-kube-api-access-2mprt\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:47.059514 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059461    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.059514 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059475    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc7ht\" (UniqueName: \"kubernetes.io/projected/6f0cdf2a-c119-4ffa-8ae0-395f249cf285-kube-api-access-tc7ht\") pod \"network-check-source-6f78d59f99-wf52v\" (UID: \"6f0cdf2a-c119-4ffa-8ae0-395f249cf285\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"
Oct 30 18:36:47.059514 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059503    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:47.059676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dl7q\" (UniqueName: \"kubernetes.io/projected/b9790e2f-d93d-47f4-aa83-6c8a54567513-kube-api-access-8dl7q\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.059676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059562    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.059676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059585    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-config-volume\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.059676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059626    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.059676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059658    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-tmp-dir\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.059819 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/334cba69-4d11-4125-9763-d79d806afbc8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:47.059819 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.059819 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz48t\" (UniqueName: \"kubernetes.io/projected/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-kube-api-access-dz48t\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.059819 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xv4l\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.059819 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh99h\" (UniqueName: \"kubernetes.io/projected/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-kube-api-access-mh99h\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.059952 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059818    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.059952 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-service-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.059952 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059880    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.059952 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.059894    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.069786 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.069765    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:36:47.069873 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.069818    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:36:47.080114 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080094    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:36:47.080114 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080104    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:36:47.080114 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080114    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080107    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080219    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080228    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080232    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080233    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080255    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080285    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:36:47.080304 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.080300    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:36:47.083113 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.083101    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:36:47.083174 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.083129    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:36:47.089415 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.089398    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:47.089415 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.089408    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:47.089525 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.089414    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:47.099611 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.099594    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4k96h\""
Oct 30 18:36:47.099705 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.099654    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kxkbv\""
Oct 30 18:36:47.100750 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.100719    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bk862\""
Oct 30 18:36:47.100831 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.100771    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:47.121468 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.121447    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pwv\" (UniqueName: \"kubernetes.io/projected/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb-kube-api-access-m7pwv\") pod \"kas-connection-checker-685895c7c9-vvw8v\" (UID: \"a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-vvw8v"
Oct 30 18:36:47.157648 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.157626    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:47.157648 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.157625    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kp722\""
Oct 30 18:36:47.157779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.157626    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mrt5r\""
Oct 30 18:36:47.157779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.157630    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:47.157779 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.157706    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hd9r2\""
Oct 30 18:36:47.160967 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.160942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8jrs\" (UniqueName: \"kubernetes.io/projected/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-kube-api-access-c8jrs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.161047 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.160972    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmtd\" (UniqueName: \"kubernetes.io/projected/c6f92791-c615-4d6e-88e8-1a3f275aa32a-kube-api-access-hhmtd\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.161047 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.160989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mprt\" (UniqueName: \"kubernetes.io/projected/80b240bd-b789-4575-8a22-f2773e4ef78d-kube-api-access-2mprt\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:47.161047 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.161047 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7ht\" (UniqueName: \"kubernetes.io/projected/6f0cdf2a-c119-4ffa-8ae0-395f249cf285-kube-api-access-tc7ht\") pod \"network-check-source-6f78d59f99-wf52v\" (UID: \"6f0cdf2a-c119-4ffa-8ae0-395f249cf285\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"
Oct 30 18:36:47.161228 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:47.161228 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dl7q\" (UniqueName: \"kubernetes.io/projected/b9790e2f-d93d-47f4-aa83-6c8a54567513-kube-api-access-8dl7q\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.161228 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.161482 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.161252    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:47.161482 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.161335    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.661303592 +0000 UTC m=+33.157195812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:47.161482 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161356    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-config-volume\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.161482 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.161482 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-tmp-dir\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.161482 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161465    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/334cba69-4d11-4125-9763-d79d806afbc8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:47.161768 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.161768 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161515    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz48t\" (UniqueName: \"kubernetes.io/projected/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-kube-api-access-dz48t\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.161768 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161669    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xv4l\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.161768 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh99h\" (UniqueName: \"kubernetes.io/projected/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-kube-api-access-mh99h\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.161768 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161729    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.161768 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-service-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161781    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-tmp-dir\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161813    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161837    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e96c52d-079e-4b90-b722-b69ad8b12730-serving-cert\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfsh\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161933    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.161979 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b240bd-b789-4575-8a22-f2773e4ef78d-cert\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:47.162361 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.161988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xgz\" (UniqueName: \"kubernetes.io/projected/1e6a72c5-3683-4fc8-8525-e82baed6a4ad-kube-api-access-46xgz\") pod \"kas-connection-checker-685895c7c9-rw5pb\" (UID: \"1e6a72c5-3683-4fc8-8525-e82baed6a4ad\") " pod="kube-system/kas-connection-checker-685895c7c9-rw5pb"
Oct 30 18:36:47.162361 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162021    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-config-volume\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.162361 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.162075    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:47.162361 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.162117    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:47.162361 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162164    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.162361 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162246    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/334cba69-4d11-4125-9763-d79d806afbc8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:47.162361 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.162346    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.162121    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls podName:c4944e38-f0de-40f8-88f3-26fc7b9c2c63 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.662107096 +0000 UTC m=+33.157999323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-djc4m" (UID: "c4944e38-f0de-40f8-88f3-26fc7b9c2c63") : secret "samples-operator-tls" not found
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162420    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-trusted-ca\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-snapshots\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162525    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjhb\" (UniqueName: \"kubernetes.io/projected/ed959a30-6e00-49b5-aceb-c543a68b9d3e-kube-api-access-ljjhb\") pod \"volume-data-source-validator-58c9f5f46d-rcf9k\" (UID: \"ed959a30-6e00-49b5-aceb-c543a68b9d3e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162560    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ttf\" (UniqueName: \"kubernetes.io/projected/8e96c52d-079e-4b90-b722-b69ad8b12730-kube-api-access-k7ttf\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162647    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162672    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.162714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162715    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnpk\" (UniqueName: \"kubernetes.io/projected/4a159864-3010-4195-84de-1407b16eb5c2-kube-api-access-pjnpk\") pod \"kas-connection-checker-685895c7c9-8cw5s\" (UID: \"4a159864-3010-4195-84de-1407b16eb5c2\") " pod="kube-system/kas-connection-checker-685895c7c9-8cw5s"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162738    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-stats-auth\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9790e2f-d93d-47f4-aa83-6c8a54567513-serving-cert\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f92791-c615-4d6e-88e8-1a3f275aa32a-serving-cert\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162960    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.162981    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fdf\" (UniqueName: \"kubernetes.io/projected/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-kube-api-access-w9fdf\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163012    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrlf\" (UniqueName: \"kubernetes.io/projected/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-kube-api-access-mhrlf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f92791-c615-4d6e-88e8-1a3f275aa32a-config\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163095    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-snapshots\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.163276 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163114    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-tmp\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.163138    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.663122187 +0000 UTC m=+33.159014393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163389    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-tmp\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163804    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-trusted-ca\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163834    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.163908    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.663884456 +0000 UTC m=+33.159776663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.163975    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-default-certificate\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.164000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.164355 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.164033    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-config\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.164786 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.164449    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:47.164786 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.164465    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:36:47.164786 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.164521    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.664506881 +0000 UTC m=+33.160399090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:36:47.164786 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.164681    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-config\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.164985 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.164885    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e96c52d-079e-4b90-b722-b69ad8b12730-serving-cert\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.164985 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.164943    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.164985 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.164950    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.165260    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.165395    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.165436    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-service-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.165585    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.665563992 +0000 UTC m=+33.161456213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.165639    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.165689    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.665672859 +0000 UTC m=+33.161565081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : secret "router-metrics-certs-default" not found
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.166409    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.166426    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.166451    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.166499    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.166663    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.166680    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.166707    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.166732    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:47.666719302 +0000 UTC m=+33.162611509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.166780    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f92791-c615-4d6e-88e8-1a3f275aa32a-config\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.166904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.166847    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.167907 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.167146    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.167907 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.167605    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9790e2f-d93d-47f4-aa83-6c8a54567513-serving-cert\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.167907 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.167678    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.168670 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.168643    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b240bd-b789-4575-8a22-f2773e4ef78d-cert\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:47.168897 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.168879    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f92791-c615-4d6e-88e8-1a3f275aa32a-serving-cert\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.168961 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.168951    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-default-certificate\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.169080 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.169061    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.169264 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.169249    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-stats-auth\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.200834 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.200814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-vvw8v"
Oct 30 18:36:47.223140 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.223105    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xgz\" (UniqueName: \"kubernetes.io/projected/1e6a72c5-3683-4fc8-8525-e82baed6a4ad-kube-api-access-46xgz\") pod \"kas-connection-checker-685895c7c9-rw5pb\" (UID: \"1e6a72c5-3683-4fc8-8525-e82baed6a4ad\") " pod="kube-system/kas-connection-checker-685895c7c9-rw5pb"
Oct 30 18:36:47.223392 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.223364    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8jrs\" (UniqueName: \"kubernetes.io/projected/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-kube-api-access-c8jrs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.225922 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.225892    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7ht\" (UniqueName: \"kubernetes.io/projected/6f0cdf2a-c119-4ffa-8ae0-395f249cf285-kube-api-access-tc7ht\") pod \"network-check-source-6f78d59f99-wf52v\" (UID: \"6f0cdf2a-c119-4ffa-8ae0-395f249cf285\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"
Oct 30 18:36:47.226212 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.226190    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfsh\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.226562 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.226505    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xv4l\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.226905 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.226847    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mprt\" (UniqueName: \"kubernetes.io/projected/80b240bd-b789-4575-8a22-f2773e4ef78d-kube-api-access-2mprt\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:47.227512 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.227443    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnpk\" (UniqueName: \"kubernetes.io/projected/4a159864-3010-4195-84de-1407b16eb5c2-kube-api-access-pjnpk\") pod \"kas-connection-checker-685895c7c9-8cw5s\" (UID: \"4a159864-3010-4195-84de-1407b16eb5c2\") " pod="kube-system/kas-connection-checker-685895c7c9-8cw5s"
Oct 30 18:36:47.228018 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.227997    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ttf\" (UniqueName: \"kubernetes.io/projected/8e96c52d-079e-4b90-b722-b69ad8b12730-kube-api-access-k7ttf\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.228808 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.228406    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjhb\" (UniqueName: \"kubernetes.io/projected/ed959a30-6e00-49b5-aceb-c543a68b9d3e-kube-api-access-ljjhb\") pod \"volume-data-source-validator-58c9f5f46d-rcf9k\" (UID: \"ed959a30-6e00-49b5-aceb-c543a68b9d3e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"
Oct 30 18:36:47.228808 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.228488    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.228808 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.228573    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmtd\" (UniqueName: \"kubernetes.io/projected/c6f92791-c615-4d6e-88e8-1a3f275aa32a-kube-api-access-hhmtd\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.228808 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.228638    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.228808 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.228708    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dl7q\" (UniqueName: \"kubernetes.io/projected/b9790e2f-d93d-47f4-aa83-6c8a54567513-kube-api-access-8dl7q\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.228808 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.228751    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrlf\" (UniqueName: \"kubernetes.io/projected/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-kube-api-access-mhrlf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.228808 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.228760    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz48t\" (UniqueName: \"kubernetes.io/projected/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-kube-api-access-dz48t\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.229236 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.229179    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fdf\" (UniqueName: \"kubernetes.io/projected/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-kube-api-access-w9fdf\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.229907 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.229888    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh99h\" (UniqueName: \"kubernetes.io/projected/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-kube-api-access-mh99h\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.233848 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.233827    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:47.254889 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.254849    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:47.274152 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.274084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"
Oct 30 18:36:47.286171 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.285863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:47.292860 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.292653    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:47.319477 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.308506    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:47.319477 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.316922    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"
Oct 30 18:36:47.506963 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.506931    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-8cw5s"
Oct 30 18:36:47.521111 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.521084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rw5pb"
Oct 30 18:36:47.591258 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.591238    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-vvw8v"]
Oct 30 18:36:47.593149 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.593121    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c1a5cc_0e5f_46ef_bc0a_e405fd732eeb.slice/crio-c33be6c60ef24e58a631561748f80157c89f51424eec89fdd51e5d9eec2a1a2e WatchSource:0}: Error finding container c33be6c60ef24e58a631561748f80157c89f51424eec89fdd51e5d9eec2a1a2e: Status 404 returned error can't find the container with id c33be6c60ef24e58a631561748f80157c89f51424eec89fdd51e5d9eec2a1a2e
Oct 30 18:36:47.632548 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.631530    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"]
Oct 30 18:36:47.634065 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.633993    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f92791_c615_4d6e_88e8_1a3f275aa32a.slice/crio-6596f80cb7c7bd1c0817d5bb20313f5345d7aeefb0e4a79548ad3b46489f2327 WatchSource:0}: Error finding container 6596f80cb7c7bd1c0817d5bb20313f5345d7aeefb0e4a79548ad3b46489f2327: Status 404 returned error can't find the container with id 6596f80cb7c7bd1c0817d5bb20313f5345d7aeefb0e4a79548ad3b46489f2327
Oct 30 18:36:47.663093 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.663077    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-sfn74"]
Oct 30 18:36:47.665213 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.665190    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9790e2f_d93d_47f4_aa83_6c8a54567513.slice/crio-2d1666c09505424df2d98850264bb43c073b50ea18afc3b310c59d853c111ee0 WatchSource:0}: Error finding container 2d1666c09505424df2d98850264bb43c073b50ea18afc3b310c59d853c111ee0: Status 404 returned error can't find the container with id 2d1666c09505424df2d98850264bb43c073b50ea18afc3b310c59d853c111ee0
Oct 30 18:36:47.669738 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.669713    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:47.669798 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.669783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:47.669833 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.669818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:47.669865 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.669854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:47.669942 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.669926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:47.669994 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.669982    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:47.670040 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.670009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.670077 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.670042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:47.670227 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670172    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:47.670227 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670223    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.670205762 +0000 UTC m=+34.166097986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : secret "router-metrics-certs-default" not found
Oct 30 18:36:47.670361 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670279    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:47.670361 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670312    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.6703005 +0000 UTC m=+34.166192718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:47.670464 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670395    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:47.670464 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670428    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls podName:c4944e38-f0de-40f8-88f3-26fc7b9c2c63 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.670417639 +0000 UTC m=+34.166309844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-djc4m" (UID: "c4944e38-f0de-40f8-88f3-26fc7b9c2c63") : secret "samples-operator-tls" not found
Oct 30 18:36:47.670565 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670477    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:47.670565 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670503    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.670494668 +0000 UTC m=+34.166386891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:47.670565 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670549    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:47.670702 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670579    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.670569224 +0000 UTC m=+34.166461433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:47.670702 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670638    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:47.670702 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670648    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:36:47.670702 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670676    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.6706667 +0000 UTC m=+34.166558906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:36:47.670904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670731    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:47.670904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670742    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:36:47.670904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670767    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.670758775 +0000 UTC m=+34.166651051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:36:47.670904 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.670793    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"]
Oct 30 18:36:47.670904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:47.670831    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:48.670820258 +0000 UTC m=+34.166712476 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:47.671811 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.671786    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded959a30_6e00_49b5_aceb_c543a68b9d3e.slice/crio-7f44bc15e73a6cf766f33ee46d9852e77ee75c82999b518ef9c720bc3b1943f4 WatchSource:0}: Error finding container 7f44bc15e73a6cf766f33ee46d9852e77ee75c82999b518ef9c720bc3b1943f4: Status 404 returned error can't find the container with id 7f44bc15e73a6cf766f33ee46d9852e77ee75c82999b518ef9c720bc3b1943f4
Oct 30 18:36:47.693024 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.692907    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mx59k"]
Oct 30 18:36:47.695024 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.695003    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e96c52d_079e_4b90_b722_b69ad8b12730.slice/crio-6853f8fb0b949d504409e0e1590c637219deedf5a0b8070bb21d423e02cbcbcc WatchSource:0}: Error finding container 6853f8fb0b949d504409e0e1590c637219deedf5a0b8070bb21d423e02cbcbcc: Status 404 returned error can't find the container with id 6853f8fb0b949d504409e0e1590c637219deedf5a0b8070bb21d423e02cbcbcc
Oct 30 18:36:47.716067 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.716043    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dhgxq"]
Oct 30 18:36:47.718096 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.718071    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b240bd_b789_4575_8a22_f2773e4ef78d.slice/crio-139df7c01e09b92f6e4b7e1f66c1994088ad5cb1f9cfcd9a16b7f58b6cab217b WatchSource:0}: Error finding container 139df7c01e09b92f6e4b7e1f66c1994088ad5cb1f9cfcd9a16b7f58b6cab217b: Status 404 returned error can't find the container with id 139df7c01e09b92f6e4b7e1f66c1994088ad5cb1f9cfcd9a16b7f58b6cab217b
Oct 30 18:36:47.724980 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.724961    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"]
Oct 30 18:36:47.725909 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.725883    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ee97fd8_b164_40ca_a9ff_e53e0f461d9e.slice/crio-bfc21e7a96ed52386bc582f793ccf51e5ef05af1a33e491a60544c19be825b8c WatchSource:0}: Error finding container bfc21e7a96ed52386bc582f793ccf51e5ef05af1a33e491a60544c19be825b8c: Status 404 returned error can't find the container with id bfc21e7a96ed52386bc582f793ccf51e5ef05af1a33e491a60544c19be825b8c
Oct 30 18:36:47.768637 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.768618    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"]
Oct 30 18:36:47.770281 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.770257    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0cdf2a_c119_4ffa_8ae0_395f249cf285.slice/crio-5e2e5b162bbebecc4eb3a01639fab0ccf23d39a84a4c22bd72d4b5cc09ea4349 WatchSource:0}: Error finding container 5e2e5b162bbebecc4eb3a01639fab0ccf23d39a84a4c22bd72d4b5cc09ea4349: Status 404 returned error can't find the container with id 5e2e5b162bbebecc4eb3a01639fab0ccf23d39a84a4c22bd72d4b5cc09ea4349
Oct 30 18:36:47.877130 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.877105    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-8cw5s"]
Oct 30 18:36:47.878865 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.878843    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a159864_3010_4195_84de_1407b16eb5c2.slice/crio-4e1f7fdbbd45cae2de375c305a08e7c7ca42fb3dba477820d7647c62af9b8463 WatchSource:0}: Error finding container 4e1f7fdbbd45cae2de375c305a08e7c7ca42fb3dba477820d7647c62af9b8463: Status 404 returned error can't find the container with id 4e1f7fdbbd45cae2de375c305a08e7c7ca42fb3dba477820d7647c62af9b8463
Oct 30 18:36:47.899395 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:47.899375    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rw5pb"]
Oct 30 18:36:47.901360 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:47.901314    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e6a72c5_3683_4fc8_8525_e82baed6a4ad.slice/crio-a32792a45e9c049059e85402f62eca5f3a15667728952dc5c1461d279f6faddf WatchSource:0}: Error finding container a32792a45e9c049059e85402f62eca5f3a15667728952dc5c1461d279f6faddf: Status 404 returned error can't find the container with id a32792a45e9c049059e85402f62eca5f3a15667728952dc5c1461d279f6faddf
Oct 30 18:36:48.265800 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.265728    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v" event={"ID":"6f0cdf2a-c119-4ffa-8ae0-395f249cf285","Type":"ContainerStarted","Data":"5e2e5b162bbebecc4eb3a01639fab0ccf23d39a84a4c22bd72d4b5cc09ea4349"}
Oct 30 18:36:48.266841 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.266817    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mx59k" event={"ID":"8e96c52d-079e-4b90-b722-b69ad8b12730","Type":"ContainerStarted","Data":"6853f8fb0b949d504409e0e1590c637219deedf5a0b8070bb21d423e02cbcbcc"}
Oct 30 18:36:48.267834 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.267807    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k" event={"ID":"ed959a30-6e00-49b5-aceb-c543a68b9d3e","Type":"ContainerStarted","Data":"7f44bc15e73a6cf766f33ee46d9852e77ee75c82999b518ef9c720bc3b1943f4"}
Oct 30 18:36:48.269047 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.269025    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vvw8v" event={"ID":"a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb","Type":"ContainerStarted","Data":"310b1c8565dd0b6dc4567c442988193bfdf5dd763b0809b153eda5a8f4ee9801"}
Oct 30 18:36:48.269119 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.269068    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-vvw8v" event={"ID":"a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb","Type":"ContainerStarted","Data":"c33be6c60ef24e58a631561748f80157c89f51424eec89fdd51e5d9eec2a1a2e"}
Oct 30 18:36:48.270245 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.270226    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8cw5s" event={"ID":"4a159864-3010-4195-84de-1407b16eb5c2","Type":"ContainerStarted","Data":"2d4c978c1b16e9e45e45165fec608a5fa2b0dc769337c172b9ee46419339688e"}
Oct 30 18:36:48.270359 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.270265    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-8cw5s" event={"ID":"4a159864-3010-4195-84de-1407b16eb5c2","Type":"ContainerStarted","Data":"4e1f7fdbbd45cae2de375c305a08e7c7ca42fb3dba477820d7647c62af9b8463"}
Oct 30 18:36:48.271270 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.271253    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb" event={"ID":"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e","Type":"ContainerStarted","Data":"bfc21e7a96ed52386bc582f793ccf51e5ef05af1a33e491a60544c19be825b8c"}
Oct 30 18:36:48.272382 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.272364    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" event={"ID":"b9790e2f-d93d-47f4-aa83-6c8a54567513","Type":"ContainerStarted","Data":"2d1666c09505424df2d98850264bb43c073b50ea18afc3b310c59d853c111ee0"}
Oct 30 18:36:48.273399 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.273381    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhgxq" event={"ID":"80b240bd-b789-4575-8a22-f2773e4ef78d","Type":"ContainerStarted","Data":"139df7c01e09b92f6e4b7e1f66c1994088ad5cb1f9cfcd9a16b7f58b6cab217b"}
Oct 30 18:36:48.278348 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.278310    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr" event={"ID":"c6f92791-c615-4d6e-88e8-1a3f275aa32a","Type":"ContainerStarted","Data":"6596f80cb7c7bd1c0817d5bb20313f5345d7aeefb0e4a79548ad3b46489f2327"}
Oct 30 18:36:48.279778 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.279756    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rw5pb" event={"ID":"1e6a72c5-3683-4fc8-8525-e82baed6a4ad","Type":"ContainerStarted","Data":"a28937cd581a83481750413566fb4ec4642ef38728baebabc642beadd0b59bca"}
Oct 30 18:36:48.279778 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.279788    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rw5pb" event={"ID":"1e6a72c5-3683-4fc8-8525-e82baed6a4ad","Type":"ContainerStarted","Data":"a32792a45e9c049059e85402f62eca5f3a15667728952dc5c1461d279f6faddf"}
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679444    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679476    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.679691    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.679821    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.679882    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.679863842 +0000 UTC m=+36.175756060 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : secret "router-metrics-certs-default" not found
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.679949    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.679983    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.679972163 +0000 UTC m=+36.175864382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680036    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680070    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls podName:c4944e38-f0de-40f8-88f3-26fc7b9c2c63 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.680057463 +0000 UTC m=+36.175949681 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-djc4m" (UID: "c4944e38-f0de-40f8-88f3-26fc7b9c2c63") : secret "samples-operator-tls" not found
Oct 30 18:36:48.680530 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680119    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680146    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.680137121 +0000 UTC m=+36.176029326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680197    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680228    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.680218547 +0000 UTC m=+36.176110766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680286    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680297    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680345    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.680316799 +0000 UTC m=+36.176209018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680419    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680428    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680456    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.680446623 +0000 UTC m=+36.176338841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:36:48.681298 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.680506    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:50.680500439 +0000 UTC m=+36.176392655 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:48.781232 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.781142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:48.791189 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.791159    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:48.881772 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.881730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:48.881956 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.881859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:48.882077 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.882060    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:48.882143 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:48.882123    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:20.882105721 +0000 UTC m=+66.377997925 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : secret "metrics-daemon-secret" not found
Oct 30 18:36:48.893936 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.893903    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:48.924400 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.901614    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:48.925087 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:48.924980    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:49.204181 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:49.202570    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-vvw8v" podStartSLOduration=309.202551738 podStartE2EDuration="5m9.202551738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:49.201525895 +0000 UTC m=+34.697418122" watchObservedRunningTime="2026-10-30 18:36:49.202551738 +0000 UTC m=+34.698443966"
Oct 30 18:36:49.204181 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:49.203240    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rw5pb" podStartSLOduration=309.203228135 podStartE2EDuration="5m9.203228135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:49.082439394 +0000 UTC m=+34.578331620" watchObservedRunningTime="2026-10-30 18:36:49.203228135 +0000 UTC m=+34.699120379"
Oct 30 18:36:49.353391 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:49.353335    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-8cw5s" podStartSLOduration=309.353307248 podStartE2EDuration="5m9.353307248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:49.322948733 +0000 UTC m=+34.818840960" watchObservedRunningTime="2026-10-30 18:36:49.353307248 +0000 UTC m=+34.849199474"
Oct 30 18:36:49.353714 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:49.353697    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k8tk8"]
Oct 30 18:36:49.393379 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:49.393348    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hktmd"]
Oct 30 18:36:50.293704 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.293664    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hktmd" event={"ID":"d7639481-1e64-49cc-a292-5cfa0cf76416","Type":"ContainerStarted","Data":"1b20c7ba1b9740fe09f9f44e8def6991f7961412588fbb6dbeb1e517d8027b0e"}
Oct 30 18:36:50.296053 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.296000    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k8tk8" event={"ID":"daacb5ca-b3fa-4344-add9-c451fc1e5e25","Type":"ContainerStarted","Data":"5e8d409944524c9be12242c109ff945511d8360e905cf992cd39d9d2f68bb1a4"}
Oct 30 18:36:50.704347 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:50.704485 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704377    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:50.704485 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:50.704485 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:50.704650 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:50.704650 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:50.704650 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704600    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:50.704650 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:50.704627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:50.704841 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.704776    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.704756264 +0000 UTC m=+40.200648482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:50.704904 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.704872    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:50.704949 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.704913    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.704900457 +0000 UTC m=+40.200792676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : secret "router-metrics-certs-default" not found
Oct 30 18:36:50.705005 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.704965    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:50.705005 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.704996    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.704987233 +0000 UTC m=+40.200879456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:50.705101 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705065    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:50.705153 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705101    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls podName:c4944e38-f0de-40f8-88f3-26fc7b9c2c63 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.705090944 +0000 UTC m=+40.200983148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-djc4m" (UID: "c4944e38-f0de-40f8-88f3-26fc7b9c2c63") : secret "samples-operator-tls" not found
Oct 30 18:36:50.705207 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705154    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:50.705207 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705185    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.705175204 +0000 UTC m=+40.201067420 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:50.705306 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705228    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:50.705306 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705250    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.705243482 +0000 UTC m=+40.201135686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:50.705306 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705298    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:50.705467 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705309    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:36:50.705467 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705353    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.705344045 +0000 UTC m=+40.201236267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:36:50.705467 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705410    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:50.705467 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705419    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:36:50.705467 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:50.705449    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:54.705437777 +0000 UTC m=+40.201329980 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:36:51.303696 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:51.303645    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k" event={"ID":"ed959a30-6e00-49b5-aceb-c543a68b9d3e","Type":"ContainerStarted","Data":"37a77cc8ad362bb50cffa74b7b5d14f5cdc1bd8d556fae5948e57f481e8126b4"}
Oct 30 18:36:51.838637 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:51.838418    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:52.183814 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:52.183752    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:52.931882 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:52.931844    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:53.188750 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.188696    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:53.316349 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.315770    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/0.log"
Oct 30 18:36:53.316349 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.315820    2574 generic.go:356] "Generic (PLEG): container finished" podID="b9790e2f-d93d-47f4-aa83-6c8a54567513" containerID="817399bb7af2587d2a1999a7a7fad51e26bf875ec19345f2c5673e7ac5e2298e" exitCode=255
Oct 30 18:36:53.316349 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.316218    2574 scope.go:122] "RemoveContainer" containerID="817399bb7af2587d2a1999a7a7fad51e26bf875ec19345f2c5673e7ac5e2298e"
Oct 30 18:36:53.316631 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.316536    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" event={"ID":"b9790e2f-d93d-47f4-aa83-6c8a54567513","Type":"ContainerDied","Data":"817399bb7af2587d2a1999a7a7fad51e26bf875ec19345f2c5673e7ac5e2298e"}
Oct 30 18:36:53.319112 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.319086    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mx59k" event={"ID":"8e96c52d-079e-4b90-b722-b69ad8b12730","Type":"ContainerStarted","Data":"1ef0bd94fb132f3f8d3384d44f6174a71b0d3c6b7ac3fc5a4473162c019e04b0"}
Oct 30 18:36:53.437271 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.437200    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k" podStartSLOduration=275.560642578 podStartE2EDuration="4m38.437186656s" totalImagesPullingTime="2.876544078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:51.427465242 +0000 UTC m=+36.923357470" watchObservedRunningTime="2026-10-30 18:36:53.437186656 +0000 UTC m=+38.933078879"
Oct 30 18:36:53.562356 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:53.561486    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-mx59k" podStartSLOduration=273.106327669 podStartE2EDuration="4m38.561466769s" totalImagesPullingTime="5.4551391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:53.559907374 +0000 UTC m=+39.055799597" watchObservedRunningTime="2026-10-30 18:36:53.561466769 +0000 UTC m=+39.057358997"
Oct 30 18:36:54.325027 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.324998    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb" event={"ID":"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e","Type":"ContainerStarted","Data":"51f3a2176c0e5fdd3e7bd71645a0b88912d40325f7aa120c15bbd342626ebad7"}
Oct 30 18:36:54.328606 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.328534    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/1.log"
Oct 30 18:36:54.330192 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.329412    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/0.log"
Oct 30 18:36:54.330192 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.329459    2574 generic.go:356] "Generic (PLEG): container finished" podID="b9790e2f-d93d-47f4-aa83-6c8a54567513" containerID="826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870" exitCode=255
Oct 30 18:36:54.330192 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.329748    2574 scope.go:122] "RemoveContainer" containerID="826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870"
Oct 30 18:36:54.330192 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.329895    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sfn74_openshift-console-operator(b9790e2f-d93d-47f4-aa83-6c8a54567513)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" podUID="b9790e2f-d93d-47f4-aa83-6c8a54567513"
Oct 30 18:36:54.330192 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.330062    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" event={"ID":"b9790e2f-d93d-47f4-aa83-6c8a54567513","Type":"ContainerDied","Data":"826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870"}
Oct 30 18:36:54.330192 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.330102    2574 scope.go:122] "RemoveContainer" containerID="817399bb7af2587d2a1999a7a7fad51e26bf875ec19345f2c5673e7ac5e2298e"
Oct 30 18:36:54.333224 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.333162    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr" event={"ID":"c6f92791-c615-4d6e-88e8-1a3f275aa32a","Type":"ContainerStarted","Data":"eb803938a8dfbb92d0a79e163f934fa5a6c2a615174440d97fc22b57eb2f677b"}
Oct 30 18:36:54.446182 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.446124    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb" podStartSLOduration=272.525854677 podStartE2EDuration="4m38.446110793s" totalImagesPullingTime="5.920256116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:54.444638168 +0000 UTC m=+39.940530393" watchObservedRunningTime="2026-10-30 18:36:54.446110793 +0000 UTC m=+39.942003020"
Oct 30 18:36:54.678103 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.675436    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr" podStartSLOduration=297.693664115 podStartE2EDuration="5m3.675416414s" totalImagesPullingTime="5.981752299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:54.570222358 +0000 UTC m=+40.066114585" watchObservedRunningTime="2026-10-30 18:36:54.675416414 +0000 UTC m=+40.171308642"
Oct 30 18:36:54.678103 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.676670    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d52ch"]
Oct 30 18:36:54.697660 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.697621    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.745634 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.745605    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d52ch"]
Oct 30 18:36:54.753158 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.753131    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:54.754146 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.754108    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:54.754313 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.754289    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5fxhm\""
Oct 30 18:36:54.757755 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:54.757848 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:54.757848 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757796    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90552f94-1a43-466e-9285-b9e00c4ed560-crio-socket\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.757848 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757834    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:54.757985 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.757985 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4njk2\" (UniqueName: \"kubernetes.io/projected/90552f94-1a43-466e-9285-b9e00c4ed560-kube-api-access-4njk2\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.757985 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:54.757985 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.757975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90552f94-1a43-466e-9285-b9e00c4ed560-data-volume\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.758127 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.758003    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:54.758127 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.758027    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:54.758127 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.758049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:54.758127 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.758069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.758127 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.758089    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:54.758279 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758198    2574 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:54.758279 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758240    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.75822698 +0000 UTC m=+48.254119184 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:54.758573 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758552    2574 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:54.758662 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758609    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls podName:c4944e38-f0de-40f8-88f3-26fc7b9c2c63 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.75859314 +0000 UTC m=+48.254485352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-djc4m" (UID: "c4944e38-f0de-40f8-88f3-26fc7b9c2c63") : secret "samples-operator-tls" not found
Oct 30 18:36:54.758662 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758657    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:54.758773 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758686    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.758675783 +0000 UTC m=+48.254567988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:54.758773 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758746    2574 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:54.758871 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758776    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.758765458 +0000 UTC m=+48.254657666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:54.758871 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758837    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:54.758871 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758844    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:36:54.758871 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758864    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.758858538 +0000 UTC m=+48.254750742 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:36:54.759059 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758906    2574 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:54.759059 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758913    2574 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:36:54.759059 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758931    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.758925606 +0000 UTC m=+48.254817809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:36:54.759059 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.758972    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.75896663 +0000 UTC m=+48.254858834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:54.759059 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.759012    2574 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:54.759059 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.759030    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.759024531 +0000 UTC m=+48.254916735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : secret "router-metrics-certs-default" not found
Oct 30 18:36:54.858858 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.858826    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.859018 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.858929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90552f94-1a43-466e-9285-b9e00c4ed560-crio-socket\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.859018 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.858962    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:54.859018 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.858991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.859183 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.859025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njk2\" (UniqueName: \"kubernetes.io/projected/90552f94-1a43-466e-9285-b9e00c4ed560-kube-api-access-4njk2\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.859183 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.859075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90552f94-1a43-466e-9285-b9e00c4ed560-data-volume\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.859183 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.859104    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:54.859183 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.859152    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.359131952 +0000 UTC m=+40.855024160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:54.859411 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.859222    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90552f94-1a43-466e-9285-b9e00c4ed560-crio-socket\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.859484 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.859459    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90552f94-1a43-466e-9285-b9e00c4ed560-data-volume\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.859599 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:54.859513    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:55.35949682 +0000 UTC m=+40.855389040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:54.923634 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.923595    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njk2\" (UniqueName: \"kubernetes.io/projected/90552f94-1a43-466e-9285-b9e00c4ed560-kube-api-access-4njk2\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:54.937439 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.937361    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-69wwp-9m4cv"]
Oct 30 18:36:54.951666 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.951632    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:54.959571 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.959543    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:54.960101 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.959601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r776j\" (UniqueName: \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:54.960101 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.959766    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:54.960101 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.959838    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:54.990670 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:54.990640    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:55.000966 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.000935    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-69wwp-9m4cv"]
Oct 30 18:36:55.009028 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.008958    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xm628\""
Oct 30 18:36:55.060676 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.060646    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.060845 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.060696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r776j\" (UniqueName: \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.060845 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.060786    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.060940 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.060844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.061485 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.061456    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.061827 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.061805    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.061924 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.061852    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.127959 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.127931    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r776j\" (UniqueName: \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.277243 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.277215    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:55.286480 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.286454    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:55.339586 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.339552    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/1.log"
Oct 30 18:36:55.340015 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.339991    2574 scope.go:122] "RemoveContainer" containerID="826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870"
Oct 30 18:36:55.340243 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:55.340219    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sfn74_openshift-console-operator(b9790e2f-d93d-47f4-aa83-6c8a54567513)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" podUID="b9790e2f-d93d-47f4-aa83-6c8a54567513"
Oct 30 18:36:55.345441 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.345417    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dhgxq" event={"ID":"80b240bd-b789-4575-8a22-f2773e4ef78d","Type":"ContainerStarted","Data":"9cf0c422eb8ad0b21cb82bf8c8d1c3483976456388c82f71951bbb0b70b6c501"}
Oct 30 18:36:55.363813 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.363787    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:55.363986 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.363970    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:55.364109 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:55.364093    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:55.364170 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:55.364159    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.364140009 +0000 UTC m=+41.860032217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:55.365558 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:55.365535    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:55.365614 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:55.365581    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:56.365567523 +0000 UTC m=+41.861459742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:55.493170 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.493097    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"]
Oct 30 18:36:55.511488 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.511464    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"]
Oct 30 18:36:55.511620 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.511594    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"
Oct 30 18:36:55.540134 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.540113    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"
Oct 30 18:36:55.555010 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.554983    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"]
Oct 30 18:36:55.555742 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.555715    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:55.568505 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.568107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnflj\" (UniqueName: \"kubernetes.io/projected/336c71cc-488f-44cc-8d22-5c8bac5f0bda-kube-api-access-nnflj\") pod \"migrator-66df5d496b-6lp98\" (UID: \"336c71cc-488f-44cc-8d22-5c8bac5f0bda\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"
Oct 30 18:36:55.568770 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.568743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqctd\" (UniqueName: \"kubernetes.io/projected/7d59ccdf-e915-4825-9055-c6b9f4c1c76d-kube-api-access-zqctd\") pod \"migrator-66df5d496b-8lkzz\" (UID: \"7d59ccdf-e915-4825-9055-c6b9f4c1c76d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"
Oct 30 18:36:55.570372 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.570350    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"]
Oct 30 18:36:55.572221 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.572202    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:55.572221 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.572173    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:55.572390 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.572345    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-88d9s\""
Oct 30 18:36:55.604750 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.602801    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dhgxq" podStartSLOduration=3.06003371 podStartE2EDuration="9.602785478s" totalImagesPullingTime="6.542751768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:55.601079005 +0000 UTC m=+41.096971234" watchObservedRunningTime="2026-10-30 18:36:55.602785478 +0000 UTC m=+41.098677706"
Oct 30 18:36:55.669616 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.669581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqctd\" (UniqueName: \"kubernetes.io/projected/7d59ccdf-e915-4825-9055-c6b9f4c1c76d-kube-api-access-zqctd\") pod \"migrator-66df5d496b-8lkzz\" (UID: \"7d59ccdf-e915-4825-9055-c6b9f4c1c76d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"
Oct 30 18:36:55.669783 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.669663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnflj\" (UniqueName: \"kubernetes.io/projected/336c71cc-488f-44cc-8d22-5c8bac5f0bda-kube-api-access-nnflj\") pod \"migrator-66df5d496b-6lp98\" (UID: \"336c71cc-488f-44cc-8d22-5c8bac5f0bda\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"
Oct 30 18:36:55.701729 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.701693    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-69wwp-9m4cv"]
Oct 30 18:36:55.703115 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:55.703089    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670b5b50_4c64_404e_b62c_9fe151aa58b2.slice/crio-41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1 WatchSource:0}: Error finding container 41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1: Status 404 returned error can't find the container with id 41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1
Oct 30 18:36:55.730475 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.730450    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqctd\" (UniqueName: \"kubernetes.io/projected/7d59ccdf-e915-4825-9055-c6b9f4c1c76d-kube-api-access-zqctd\") pod \"migrator-66df5d496b-8lkzz\" (UID: \"7d59ccdf-e915-4825-9055-c6b9f4c1c76d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"
Oct 30 18:36:55.730967 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.730951    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnflj\" (UniqueName: \"kubernetes.io/projected/336c71cc-488f-44cc-8d22-5c8bac5f0bda-kube-api-access-nnflj\") pod \"migrator-66df5d496b-6lp98\" (UID: \"336c71cc-488f-44cc-8d22-5c8bac5f0bda\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"
Oct 30 18:36:55.771467 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.771241    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:55.848970 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.848949    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"
Oct 30 18:36:55.865300 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:55.865275    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"
Oct 30 18:36:56.006119 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.006087    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:56.279247 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.279215    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"]
Oct 30 18:36:56.280294 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:56.280264    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336c71cc_488f_44cc_8d22_5c8bac5f0bda.slice/crio-b84c7fdfd0ff925d3a6890b81d3ec70a4aaca3768a37e4ef9839e0b4475b0608 WatchSource:0}: Error finding container b84c7fdfd0ff925d3a6890b81d3ec70a4aaca3768a37e4ef9839e0b4475b0608: Status 404 returned error can't find the container with id b84c7fdfd0ff925d3a6890b81d3ec70a4aaca3768a37e4ef9839e0b4475b0608
Oct 30 18:36:56.335579 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.335552    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"]
Oct 30 18:36:56.336697 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:56.336670    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d59ccdf_e915_4825_9055_c6b9f4c1c76d.slice/crio-271d8b24c9d5a4231742055df0db674e938ba0d77a8ac519038722a9c8e2ee1d WatchSource:0}: Error finding container 271d8b24c9d5a4231742055df0db674e938ba0d77a8ac519038722a9c8e2ee1d: Status 404 returned error can't find the container with id 271d8b24c9d5a4231742055df0db674e938ba0d77a8ac519038722a9c8e2ee1d
Oct 30 18:36:56.350053 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.350025    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz" event={"ID":"7d59ccdf-e915-4825-9055-c6b9f4c1c76d","Type":"ContainerStarted","Data":"271d8b24c9d5a4231742055df0db674e938ba0d77a8ac519038722a9c8e2ee1d"}
Oct 30 18:36:56.351835 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.351813    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hktmd" event={"ID":"d7639481-1e64-49cc-a292-5cfa0cf76416","Type":"ContainerStarted","Data":"81856373b3a1c422d92b43b194ed6b5cae1867d9d558a1f4fc80a60feb63c854"}
Oct 30 18:36:56.351936 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.351923    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:56.353467 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.353439    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-69wwp-9m4cv" event={"ID":"670b5b50-4c64-404e-b62c-9fe151aa58b2","Type":"ContainerStarted","Data":"dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5"}
Oct 30 18:36:56.353569 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.353480    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-69wwp-9m4cv" event={"ID":"670b5b50-4c64-404e-b62c-9fe151aa58b2","Type":"ContainerStarted","Data":"41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1"}
Oct 30 18:36:56.354647 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.354619    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98" event={"ID":"336c71cc-488f-44cc-8d22-5c8bac5f0bda","Type":"ContainerStarted","Data":"b84c7fdfd0ff925d3a6890b81d3ec70a4aaca3768a37e4ef9839e0b4475b0608"}
Oct 30 18:36:56.356458 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.356435    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v" event={"ID":"6f0cdf2a-c119-4ffa-8ae0-395f249cf285","Type":"ContainerStarted","Data":"ab1b8eb860c809011573686898b295518dd89c3ce5e25f5af95ee2db5d44ea63"}
Oct 30 18:36:56.382673 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.382646    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:56.382789 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.382776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:56.382890 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:56.382871    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:56.382957 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:56.382923    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.38290779 +0000 UTC m=+43.878800010 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:56.383608 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:56.383529    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 18:36:56.383608 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:56.383585    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:58.3835705 +0000 UTC m=+43.879462717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : configmap "kube-rbac-proxy" not found
Oct 30 18:36:56.471344 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.471277    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hktmd" podStartSLOduration=35.05144643 podStartE2EDuration="41.471261187s" totalImagesPullingTime="6.419814757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.470562096 +0000 UTC m=+41.966454323" watchObservedRunningTime="2026-10-30 18:36:56.471261187 +0000 UTC m=+41.967153412"
Oct 30 18:36:56.552715 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.552630    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vg6tg"]
Oct 30 18:36:56.586756 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.585820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.591920 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.591866    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v" podStartSLOduration=272.128558228 podStartE2EDuration="4m39.591848126s" totalImagesPullingTime="7.463289898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.590710114 +0000 UTC m=+42.086602340" watchObservedRunningTime="2026-10-30 18:36:56.591848126 +0000 UTC m=+42.087740369"
Oct 30 18:36:56.608988 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.608960    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vg6tg"]
Oct 30 18:36:56.657960 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.657930    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:36:56.657960 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.657954    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:36:56.658194 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.657967    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:36:56.658194 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.657977    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:36:56.658194 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.657940    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bgghq\""
Oct 30 18:36:56.684970 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.684945    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r248d\" (UniqueName: \"kubernetes.io/projected/1a705f33-1517-4f85-8eca-10395ff58b12-kube-api-access-r248d\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.684970 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.684982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a705f33-1517-4f85-8eca-10395ff58b12-signing-key\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.685157 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.685096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a705f33-1517-4f85-8eca-10395ff58b12-signing-cabundle\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.716816 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.716757    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-69wwp-9m4cv" podStartSLOduration=2.7167420890000002 podStartE2EDuration="2.716742089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:56.715609828 +0000 UTC m=+42.211502079" watchObservedRunningTime="2026-10-30 18:36:56.716742089 +0000 UTC m=+42.212634349"
Oct 30 18:36:56.786442 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.786400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r248d\" (UniqueName: \"kubernetes.io/projected/1a705f33-1517-4f85-8eca-10395ff58b12-kube-api-access-r248d\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.786442 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.786446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a705f33-1517-4f85-8eca-10395ff58b12-signing-key\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.786672 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.786491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a705f33-1517-4f85-8eca-10395ff58b12-signing-cabundle\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.787295 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.787260    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a705f33-1517-4f85-8eca-10395ff58b12-signing-cabundle\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.789240 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.789218    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a705f33-1517-4f85-8eca-10395ff58b12-signing-key\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.850077 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.850008    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r248d\" (UniqueName: \"kubernetes.io/projected/1a705f33-1517-4f85-8eca-10395ff58b12-kube-api-access-r248d\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:56.861843 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.861817    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:56.912280 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:56.912238    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:57.063131 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:57.063100    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:57.255399 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:57.255316    2574 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:57.255546 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:57.255428    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:57.255781 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:57.255769    2574 scope.go:122] "RemoveContainer" containerID="826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870"
Oct 30 18:36:57.255967 ip-10-0-129-88 kubenswrapper[2574]: E1030 18:36:57.255950    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sfn74_openshift-console-operator(b9790e2f-d93d-47f4-aa83-6c8a54567513)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" podUID="b9790e2f-d93d-47f4-aa83-6c8a54567513"
Oct 30 18:36:57.308869 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:57.308849    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vg6tg"]
Oct 30 18:36:57.433433 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:57.433390    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:57.436203 ip-10-0-129-88 kubenswrapper[2574]: W1030 18:36:57.436174    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a705f33_1517_4f85_8eca_10395ff58b12.slice/crio-e78e1e0874181c9ac5126599740a232797b7e2b873614e291dc812a53cc9474d WatchSource:0}: Error finding container e78e1e0874181c9ac5126599740a232797b7e2b873614e291dc812a53cc9474d: Status 404 returned error can't find the container with id e78e1e0874181c9ac5126599740a232797b7e2b873614e291dc812a53cc9474d
Oct 30 18:36:57.713785 ip-10-0-129-88 kubenswrapper[2574]: I1030 18:36:57.713751    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:57.850410 ip-10-0-129-88 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 18:36:57.856962 ip-10-0-129-88 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 18:36:57.857146 ip-10-0-129-88 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 18:36:57.857204 ip-10-0-129-88 systemd[1]: kubelet.service: Consumed 3.162s CPU time.
Oct 30 18:36:57.875849 ip-10-0-129-88 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 18:36:57.956478 ip-10-0-129-88 kubenswrapper[5025]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:57.956478 ip-10-0-129-88 kubenswrapper[5025]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 18:36:57.956478 ip-10-0-129-88 kubenswrapper[5025]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:57.956478 ip-10-0-129-88 kubenswrapper[5025]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 18:36:57.960205 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960090    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.960205 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960203    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.960205 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960208    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.960205 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960213    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960218    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960223    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960227    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960231    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960235    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960240    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960244    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960249    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960253    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960257    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960262    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960266    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960270    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960274    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960279    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960283    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960286    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960290    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960295    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.960491 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960299    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960303    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960307    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960311    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960316    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960334    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960338    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960342    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960351    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960356    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960360    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960364    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960368    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960372    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960376    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960381    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960385    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960389    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960393    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960397    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.961375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960401    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960405    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960409    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960413    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960417    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960422    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960426    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960430    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960434    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960439    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960443    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960447    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960451    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960456    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960460    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960466    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960470    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960475    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960480    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.962244 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960494    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960499    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960504    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960508    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960512    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960516    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960520    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960524    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960529    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960533    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960537    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960542    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960548    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960554    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960559    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960564    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960568    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960572    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960576    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.963069 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960580    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960584    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960589    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960594    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960598    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960601    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960606    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960610    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960614    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960619    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960623    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960627    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960631    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960636    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960640    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960644    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960648    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960652    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960656    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.963907 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960660    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960664    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960668    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960673    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960677    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.960684    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961898    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961908    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961914    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961919    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961924    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961929    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961933    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961937    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961941    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961946    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961954    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961958    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961962    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.964746 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961968    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961972    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961976    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961980    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961984    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961990    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.961998    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962002    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962007    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962011    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962015    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962019    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962023    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962028    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962032    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962036    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962040    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962045    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962049    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962053    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.965580 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962057    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962061    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962065    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962069    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962073    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962077    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962081    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962086    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962090    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962094    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962100    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962105    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962109    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962113    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962117    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962122    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962126    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962129    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962134    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.966453 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962139    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962143    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962147    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962151    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962156    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962161    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962164    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962168    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962172    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962177    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962181    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962185    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962189    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962194    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962198    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962202    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962206    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962210    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962214    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962218    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.967296 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962223    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962227    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962232    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962236    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962241    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962246    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962250    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962254    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962258    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962263    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962267    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962271    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962275    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962279    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962284    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962288    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962292    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962296    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962300    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962304    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.968173 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962308    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962313    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962317    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962336    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962340    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962344    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962348    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962352    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962359    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962365    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962371    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962376    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962381    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.962385    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963146    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963153    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963163    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963168    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963173    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.969047 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963177    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963181    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963186    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963190    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963194    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963198    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963203    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963207    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963211    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963215    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963219    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963224    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963228    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963232    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963237    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963241    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963245    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963249    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963254    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963259    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.969963 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963263    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963267    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963271    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963276    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963280    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963285    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963289    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963293    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963297    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963302    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963307    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963311    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963332    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963339    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963345    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963349    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963353    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963358    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963362    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.970833 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963367    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963371    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963375    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963380    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963383    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963388    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963392    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963396    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963401    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963405    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963409    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963413    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963418    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963422    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963426    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963431    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963435    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963439    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963443    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963447    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.971651 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963451    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963456    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963460    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963464    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963469    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963473    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963478    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963482    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963487    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963491    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963495    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963499    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963503    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963507    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963511    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963515    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963519    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963524    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963528    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963532    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.972657 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963536    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963541    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963545    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963549    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963553    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963557    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963561    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963565    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963569    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963573    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963577    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963581    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963586    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963590    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963594    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963598    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963602    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963606    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963611    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.973524 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963618    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963623    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.963628    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963762    5025 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963772    5025 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963781    5025 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963788    5025 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963794    5025 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963799    5025 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963806    5025 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963812    5025 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963817    5025 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963822    5025 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963828    5025 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963833    5025 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963838    5025 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963842    5025 flags.go:64] FLAG: --cgroup-root=""
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963847    5025 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963852    5025 flags.go:64] FLAG: --client-ca-file=""
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963856    5025 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963861    5025 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963868    5025 flags.go:64] FLAG: --cluster-domain=""
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963873    5025 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963879    5025 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 18:36:57.974364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963884    5025 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963889    5025 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963896    5025 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963900    5025 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963905    5025 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963910    5025 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963915    5025 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963925    5025 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963930    5025 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963935    5025 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963940    5025 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963947    5025 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963952    5025 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963957    5025 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963962    5025 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963967    5025 flags.go:64] FLAG: --enable-server="true"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963972    5025 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963978    5025 flags.go:64] FLAG: --event-burst="100"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963983    5025 flags.go:64] FLAG: --event-qps="50"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963988    5025 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963993    5025 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.963998    5025 flags.go:64] FLAG: --eviction-hard=""
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964005    5025 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964009    5025 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964015    5025 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 18:36:57.975389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964020    5025 flags.go:64] FLAG: --eviction-soft=""
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964025    5025 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964030    5025 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964034    5025 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964039    5025 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964044    5025 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964048    5025 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964053    5025 flags.go:64] FLAG: --feature-gates=""
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964084    5025 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964090    5025 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964096    5025 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964102    5025 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964107    5025 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964112    5025 flags.go:64] FLAG: --help="false"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964119    5025 flags.go:64] FLAG: --hostname-override="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964125    5025 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964130    5025 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964135    5025 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964140    5025 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964146    5025 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964150    5025 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964155    5025 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964159    5025 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 18:36:57.976557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964164    5025 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964169    5025 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964174    5025 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964178    5025 flags.go:64] FLAG: --kube-reserved=""
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964183    5025 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964187    5025 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964192    5025 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964197    5025 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964201    5025 flags.go:64] FLAG: --lock-file=""
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964206    5025 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964210    5025 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964215    5025 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964223    5025 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964228    5025 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964233    5025 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964238    5025 flags.go:64] FLAG: --logging-format="text"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964243    5025 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964248    5025 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964252    5025 flags.go:64] FLAG: --manifest-url=""
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964257    5025 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964263    5025 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964268    5025 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964274    5025 flags.go:64] FLAG: --max-pods="110"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964281    5025 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964285    5025 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 18:36:57.977573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964291    5025 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964296    5025 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964301    5025 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964305    5025 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964310    5025 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964335    5025 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964341    5025 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964345    5025 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964350    5025 flags.go:64] FLAG: --pod-cidr=""
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964355    5025 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964359    5025 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964365    5025 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964369    5025 flags.go:64] FLAG: --port="10250"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964374    5025 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964378    5025 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0586cbc82d324dc74"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964383    5025 flags.go:64] FLAG: --qos-reserved=""
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964388    5025 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964393    5025 flags.go:64] FLAG: --register-node="true"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964397    5025 flags.go:64] FLAG: --register-with-taints=""
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964403    5025 flags.go:64] FLAG: --registry-burst="10"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964408    5025 flags.go:64] FLAG: --registry-qps="5"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964412    5025 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964417    5025 flags.go:64] FLAG: --reserved-memory=""
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964422    5025 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964427    5025 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964432    5025 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 18:36:57.978219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964436    5025 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964441    5025 flags.go:64] FLAG: --runonce="false"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964446    5025 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964451    5025 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964457    5025 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964462    5025 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964466    5025 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964472    5025 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964477    5025 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964482    5025 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964486    5025 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964491    5025 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964495    5025 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964500    5025 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964504    5025 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964509    5025 flags.go:64] FLAG: --system-cgroups=""
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964514    5025 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964522    5025 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964527    5025 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964531    5025 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964538    5025 flags.go:64] FLAG: --tls-min-version=""
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964542    5025 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964547    5025 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964551    5025 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964556    5025 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 18:36:57.979146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964561    5025 flags.go:64] FLAG: --v="2"
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964567    5025 flags.go:64] FLAG: --version="false"
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964573    5025 flags.go:64] FLAG: --vmodule=""
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964579    5025 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.964585    5025 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964783    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964788    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964793    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964798    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964802    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964806    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964813    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964817    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964822    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964826    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964834    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964839    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964843    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964848    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964852    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964857    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.980172 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964861    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964865    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964870    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964874    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964878    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964882    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964887    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964893    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964898    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964903    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964907    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964911    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964916    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964920    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964924    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964928    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964933    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964937    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964942    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964946    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.981004 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964950    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964954    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964959    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964964    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964968    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964972    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964976    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964981    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964985    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964989    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964993    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.964998    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965002    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965006    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965010    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965014    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965018    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965022    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965026    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.981805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965031    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965035    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965039    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965043    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965047    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965051    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965055    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965059    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965063    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965067    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965071    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965075    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965079    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965083    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965089    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965095    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965099    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965103    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965107    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965112    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.982519 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965116    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965121    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965125    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965129    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965133    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965137    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965141    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965145    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965149    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965154    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965158    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965163    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965167    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965171    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965175    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965191    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965195    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965199    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965203    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.983159 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965208    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965212    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965216    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965220    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965223    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965227    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965232    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965237    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965243    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965247    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965253    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965259    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.965297    5025 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965372    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965379    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965384    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.983768 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965388    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965393    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965397    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965401    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965405    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965410    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965414    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965418    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965422    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965426    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965430    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965434    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965438    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965443    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965447    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965451    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965455    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965462    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965467    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.984249 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965471    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965475    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965480    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965484    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965490    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965495    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965501    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965505    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965509    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965514    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965518    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965522    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965527    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965538    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965542    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965547    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965551    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965555    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965559    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965564    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.984867 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965568    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965572    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965576    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965583    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965589    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965594    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965599    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965603    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965607    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965611    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965616    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965620    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965624    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965629    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965633    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965637    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965643    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965647    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965653    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.985452 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965657    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965661    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965665    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965670    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965674    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965678    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965683    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965687    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965692    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965696    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965700    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965704    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965708    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965712    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965716    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965720    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965725    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965730    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965734    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.986059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965739    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965743    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965747    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965751    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965755    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965759    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965763    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965767    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965772    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965776    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965782    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965786    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965792    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965796    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965800    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965805    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965809    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965813    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965817    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.986714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965822    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965826    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965831    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965835    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965839    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965844    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.965848    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.970780    5025 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.970798    5025 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.970820    5025 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.970828    5025 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970896    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970902    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970905    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970909    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970912    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970915    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970918    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970920    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.987352 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970923    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970927    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970930    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970932    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970935    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970938    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970941    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970943    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970946    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970949    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970953    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970957    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970960    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970963    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970966    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970968    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970971    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970974    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970976    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.988059 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970979    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970982    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970985    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970988    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970991    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970994    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.970998    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971003    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971007    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971012    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971016    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971020    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971024    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971028    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971032    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971037    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971041    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971044    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971046    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.988713 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971049    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971053    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971057    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971060    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971064    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971067    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971070    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971072    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971075    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971078    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971080    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971084    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971087    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971090    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971092    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971095    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971098    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971101    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971104    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.989375 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971106    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971109    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971112    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971114    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971117    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971119    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971122    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971124    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971127    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971129    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971132    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971134    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971137    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971139    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971142    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971147    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971152    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971156    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971160    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971164    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.989959 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971168    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971172    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971175    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971180    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971184    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971189    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971193    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971197    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971202    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971207    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971211    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971215    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971220    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971225    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971230    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971234    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971239    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971244    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971247    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971250    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.990659 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971252    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.971281    5025 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971466    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971471    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971474    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971477    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971480    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971483    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971486    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971488    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971491    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971494    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971496    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971499    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971501    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971504    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.991251 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971507    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971510    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971512    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971515    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971518    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971520    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971523    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971527    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971531    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971535    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971539    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971544    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971547    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971551    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971556    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971560    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971564    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971568    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971573    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.991780 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971577    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971579    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971582    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971586    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971590    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971594    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971598    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971602    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971605    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971607    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971611    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971615    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971618    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971621    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971624    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971626    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971629    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971632    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971634    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971639    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.992341 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971642    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971646    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971648    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971651    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971654    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971656    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971659    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971662    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971666    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971670    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971681    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971686    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971690    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971692    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971695    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971697    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971700    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971703    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971706    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971711    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.992917 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971715    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971719    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971722    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971726    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971731    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971735    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971739    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971744    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971748    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971753    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971757    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971762    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971767    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971771    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971775    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971780    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971783    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971786    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971788    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.993467 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971791    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971794    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971796    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971799    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971802    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971804    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971807    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971827    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971832    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971837    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971844    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971851    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971854    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971857    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.971885    5025 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971937    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 18:36:57.994131 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971944    5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971947    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971951    5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971954    5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971957    5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971960    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971963    5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971966    5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971968    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971971    5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971973    5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971976    5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971980    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971984    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971988    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971992    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971996    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.971999    5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972002    5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 18:36:57.994714 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972006    5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972010    5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972013    5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972016    5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972019    5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972022    5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972024    5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972029    5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972032    5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972035    5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972037    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972040    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972042    5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972045    5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972047    5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972050    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972052    5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972055    5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972058    5025 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972061    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 18:36:57.995302 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972066    5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972070    5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972074    5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972079    5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972082    5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972084    5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972087    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972090    5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972093    5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972095    5025 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972098    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972100    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972104    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972106    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972110    5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972114    5025 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972118    5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972122    5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972126    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 18:36:57.996033 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972131    5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972135    5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972139    5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972143    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972147    5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972151    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972155    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972159    5025 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972165    5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972171    5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972176    5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972179    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972181    5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972184    5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972186    5025 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972189    5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972191    5025 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972194    5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972196    5025 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972199    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 18:36:57.996683 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972202    5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972205    5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972207    5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972211    5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972215    5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972219    5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972224    5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972227    5025 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972230    5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972233    5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972235    5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972238    5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972242    5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972245    5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972247    5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972250    5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972253    5025 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972255    5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972260    5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972262    5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 18:36:57.997404 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972265    5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972267    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972270    5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972274    5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972278    5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972282    5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:36:57.972286    5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.972372    5025 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.973466    5025 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.973527    5025 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.973749    5025 server.go:1014] "Starting client certificate rotation"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.973878    5025 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.973921    5025 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 18:31:15 +0000 UTC" deadline="2028-06-02 11:10:11.40253129 +0000 UTC"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.973968    5025 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13936h33m13.428566859s"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.974389    5025 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.975618    5025 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 18:36:57.997943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.978876    5025 log.go:25] "Validated CRI v1 runtime API"
Oct 30 18:36:57.998363 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.981013    5025 log.go:25] "Validated CRI v1 image API"
Oct 30 18:36:57.998363 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.981745    5025 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 18:36:57.998363 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.984078    5025 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 879b331d-b122-4565-a679-eb9df7d20076:/dev/nvme0n1p3 b90366e3-eb25-410e-aeb2-6e7fb747acfe:/dev/nvme0n1p4]
Oct 30 18:36:57.998483 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.984102    5025 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/139df7c01e09b92f6e4b7e1f66c1994088ad5cb1f9cfcd9a16b7f58b6cab217b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/139df7c01e09b92f6e4b7e1f66c1994088ad5cb1f9cfcd9a16b7f58b6cab217b/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ac122194e6e5df57c31b21a993a4ec0409a7bbafd0c1ffc6143d01e9d00d789/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ac122194e6e5df57c31b21a993a4ec0409a7bbafd0c1ffc6143d01e9d00d789/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b20c7ba1b9740fe09f9f44e8def6991f7961412588fbb6dbeb1e517d8027b0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b20c7ba1b9740fe09f9f44e8def6991f7961412588fbb6dbeb1e517d8027b0e/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/271d8b24c9d5a4231742055df0db674e938ba0d77a8ac519038722a9c8e2ee1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/271d8b24c9d5a4231742055df0db674e938ba0d77a8ac519038722a9c8e2ee1d/userdata/shm major:0 minor:344 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d1666c09505424df2d98850264bb43c073b50ea18afc3b310c59d853c111ee0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d1666c09505424df2d98850264bb43c073b50ea18afc3b310c59d853c111ee0/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d79841a1a769d7f8cdd3485314cedb0af23031618e37da1c66bb9d8b2391c2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d79841a1a769d7f8cdd3485314cedb0af23031618e37da1c66bb9d8b2391c2d/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1/userdata/shm major:0 minor:335 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e1f7fdbbd45cae2de375c305a08e7c7ca42fb3dba477820d7647c62af9b8463/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e1f7fdbbd45cae2de375c305a08e7c7ca42fb3dba477820d7647c62af9b8463/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e2e5b162bbebecc4eb3a01639fab0ccf23d39a84a4c22bd72d4b5cc09ea4349/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e2e5b162bbebecc4eb3a01639fab0ccf23d39a84a4c22bd72d4b5cc09ea4349/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e8d409944524c9be12242c109ff945511d8360e905cf992cd39d9d2f68bb1a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e8d409944524c9be12242c109ff945511d8360e905cf992cd39d9d2f68bb1a4/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6596f80cb7c7bd1c0817d5bb20313f5345d7aeefb0e4a79548ad3b46489f2327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6596f80cb7c7bd1c0817d5bb20313f5345d7aeefb0e4a79548ad3b46489f2327/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6853f8fb0b949d504409e0e1590c637219deedf5a0b8070bb21d423e02cbcbcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6853f8fb0b949d504409e0e1590c637219deedf5a0b8070bb21d423e02cbcbcc/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e0626d39502c3fc05873906b92be1a6ef36223010e13cdfa69152ae3e504482/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e0626d39502c3fc05873906b92be1a6ef36223010e13cdfa69152ae3e504482/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f44bc15e73a6cf766f33ee46d9852e77ee75c82999b518ef9c720bc3b1943f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f44bc15e73a6cf766f33ee46d9852e77ee75c82999b518ef9c720bc3b1943f4/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e0ff618735547c8ea5cf78e9c01829e04b0d9871061958c101fd654355b8203/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e0ff618735547c8ea5cf78e9c01829e04b0d9871061958c101fd654355b8203/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a32792a45e9c049059e85402f62eca5f3a15667728952dc5c1461d279f6faddf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a32792a45e9c049059e85402f62eca5f3a15667728952dc5c1461d279f6faddf/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b84c7fdfd0ff925d3a6890b81d3ec70a4aaca3768a37e4ef9839e0b4475b0608/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b84c7fdfd0ff925d3a6890b81d3ec70a4aaca3768a37e4ef9839e0b4475b0608/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bfc21e7a96ed52386bc582f793ccf51e5ef05af1a33e491a60544c19be825b8c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bfc21e7a96ed52386bc582f793ccf51e5ef05af1a33e491a60544c19be825b8c/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c33be6c60ef24e58a631561748f80157c89f51424eec89fdd51e5d9eec2a1a2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c33be6c60ef24e58a631561748f80157c89f51424eec89fdd51e5d9eec2a1a2e/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e292958ee020bc49993fbd96dc254942c8ac03afeaf637d1c24b6989d9d6582d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e292958ee020bc49993fbd96dc254942c8ac03afeaf637d1c24b6989d9d6582d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3c70ee516a89708a242cefbbf143a08d3e24beb903c1b1b14245c0441e1c7f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3c70ee516a89708a242cefbbf143a08d3e24beb903c1b1b14245c0441e1c7f3/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e47c8cbc0af923b5cde2c263466bb43fd9b4f23e12ead943f5c8670666440005/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e47c8cbc0af923b5cde2c263466bb43fd9b4f23e12ead943f5c8670666440005/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e63e724d48f0f19b7214cc23c6af655ca1f287b1ee37570f91201c4145de257d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e63e724d48f0f19b7214cc23c6af655ca1f287b1ee37570f91201c4145de257d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e78e1e0874181c9ac5126599740a232797b7e2b873614e291dc812a53cc9474d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e78e1e0874181c9ac5126599740a232797b7e2b873614e291dc812a53cc9474d/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~projected/kube-api-access-ddgrz:{mountpoint:/var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~projected/kube-api-access-ddgrz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a705f33-1517-4f85-8eca-10395ff58b12/volumes/kubernetes.io~projected/kube-api-access-r248d:{mountpoint:/var/lib/kubelet/pods/1a705f33-1517-4f85-8eca-10395ff58b12/volumes/kubernetes.io~projected/kube-api-access-r248d major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a705f33-1517-4f85-8eca-10395ff58b12/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/1a705f33-1517-4f85-8eca-10395ff58b12/volumes/kubernetes.io~secret/signing-key major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e6a72c5-3683-4fc8-8525-e82baed6a4ad/volumes/kubernetes.io~projected/kube-api-access-46xgz:{mountpoint:/var/lib/kubelet/pods/1e6a72c5-3683-4fc8-8525-e82baed6a4ad/volumes/kubernetes.io~projected/kube-api-access-46xgz major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/336c71cc-488f-44cc-8d22-5c8bac5f0bda/volumes/kubernetes.io~projected/kube-api-access-nnflj:{mountpoint:/var/lib/kubelet/pods/336c71cc-488f-44cc-8d22-5c8bac5f0bda/volumes/kubernetes.io~projected/kube-api-access-nnflj major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e/volumes/kubernetes.io~projected/kube-api-access-mhrlf:{mountpoint:/var/lib/kubelet/pods/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e/volumes/kubernetes.io~projected/kube-api-access-mhrlf major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~projected/kube-api-access-c8jrs:{mountpoint:/var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~projected/kube-api-access-c8jrs major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~secret/default-certificate major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~secret/stats-auth major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a159864-3010-4195-84de-1407b16eb5c2/volumes/kubernetes.io~projected/kube-api-access-pjnpk:{mountpoint:/var/lib/kubelet/pods/4a159864-3010-4195-84de-1407b16eb5c2/volumes/kubernetes.io~projected/kube-api-access-pjnpk major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/511cf18c-8d23-40e7-9e68-f683e3fe9e7d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/511cf18c-8d23-40e7-9e68-f683e3fe9e7d/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/670b5b50-4c64-404e-b62c-9fe151aa58b2/volumes/kubernetes.io~projected/kube-api-access-r776j:{mountpoint:/var/lib/kubelet/pods/670b5b50-4c64-404e-b62c-9fe151aa58b2/volumes/kubernetes.io~projected/kube-api-access-r776j major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f0cdf2a-c119-4ffa-8ae0-395f249cf285/volumes/kubernetes.io~projected/kube-api-access-tc7ht:{mountpoint:/var/lib/kubelet/pods/6f0cdf2a-c119-4ffa-8ae0-395f249cf285/volumes/kubernetes.io~projected/kube-api-access-tc7ht major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77bb5229-a381-4bf0-82ee-e4eb959cf7eb/volumes/kubernetes.io~projected/kube-api-access-n59hl:{mountpoint:/var/lib/kubelet/pods/77bb5229-a381-4bf0-82ee-e4eb959cf7eb/volumes/kubernetes.io~projected/kube-api-access-n59hl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d59ccdf-e915-4825-9055-c6b9f4c1c76d/volumes/kubernetes.io~projected/kube-api-access-zqctd:{mountpoint:/var/lib/kubelet/pods/7d59ccdf-e915-4825-9055-c6b9f4c1c76d/volumes/kubernetes.io~projected/kube-api-access-zqctd major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80b240bd-b789-4575-8a22-f2773e4ef78d/volumes/kubernetes.io~projected/kube-api-access-2mprt:{mountpoint:/var/lib/kubelet/pods/80b240bd-b789-4575-8a22-f2773e4ef78d/volumes/kubernetes.io~projected/kube-api-access-2mprt major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80b240bd-b789-4575-8a22-f2773e4ef78d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/80b240bd-b789-4575-8a22-f2773e4ef78d/volumes/kubernetes.io~secret/cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~projected/kube-api-access-5xv4l:{mountpoint:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~projected/kube-api-access-5xv4l major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e96c52d-079e-4b90-b722-b69ad8b12730/volumes/kubernetes.io~projected/kube-api-access-k7ttf:{mountpoint:/var/lib/kubelet/pods/8e96c52d-079e-4b90-b722-b69ad8b12730/volumes/kubernetes.io~projected/kube-api-access-k7ttf major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e96c52d-079e-4b90-b722-b69ad8b12730/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8e96c52d-079e-4b90-b722-b69ad8b12730/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90552f94-1a43-466e-9285-b9e00c4ed560/volumes/kubernetes.io~projected/kube-api-access-4njk2:{mountpoint:/var/lib/kubelet/pods/90552f94-1a43-466e-9285-b9e00c4ed560/volumes/kubernetes.io~projected/kube-api-access-4njk2 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0/volumes/kubernetes.io~projected/kube-api-access-r6fmf:{mountpoint:/var/lib/kubelet/pods/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0/volumes/kubernetes.io~projected/kube-api-access-r6fmf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d51c59f-38ab-4311-bf40-cf79edf28d4a/volumes/kubernetes.io~projected/kube-api-access-9d82m:{mountpoint:/var/lib/kubelet/pods/9d51c59f-38ab-4311-bf40-cf79edf28d4a/volumes/kubernetes.io~projected/kube-api-access-9d82m major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a56de33d-a1ed-44e6-863f-11ae45cec41f/volumes/kubernetes.io~projected/kube-api-access-95vb9:{mountpoint:/var/lib/kubelet/pods/a56de33d-a1ed-44e6-863f-11ae45cec41f/volumes/kubernetes.io~projected/kube-api-access-95vb9 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb/volumes/kubernetes.io~projected/kube-api-access-m7pwv:{mountpoint:/var/lib/kubelet/pods/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb/volumes/kubernetes.io~projected/kube-api-access-m7pwv major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/acca8ca1-d447-4a1f-bd99-8dc48014e033/volumes/kubernetes.io~projected/kube-api-access-5x95b:{mountpoint:/var/lib/kubelet/pods/acca8ca1-d447-4a1f-bd99-8dc48014e033/volumes/kubernetes.io~projected/kube-api-access-5x95b major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/accf98f1-25fe-4fb4-b63a-05ad93a65d1f/volumes/kubernetes.io~projected/kube-api-access-mh99h:{mountpoint:/var/lib/kubelet/pods/accf98f1-25fe-4fb4-b63a-05ad93a65d1f/volumes/kubernetes.io~projected/kube-api-access-mh99h major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volumes/kubernetes.io~projected/kube-api-access-k58pb:{mountpoint:/var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volumes/kubernetes.io~projected/kube-api-access-k58pb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9790e2f-d93d-47f4-aa83-6c8a54567513/volumes/kubernetes.io~projected/kube-api-access-8dl7q:{mountpoint:/var/lib/kubelet/pods/b9790e2f-d93d-47f4-aa83-6c8a54567513/volumes/kubernetes.io~projected/kube-api-access-8dl7q major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9790e2f-d93d-47f4-aa83-6c8a54567513/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b9790e2f-d93d-47f4-aa83-6c8a54567513/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4944e38-f0de-40f8-88f3-26fc7b9c2c63/volumes/kubernetes.io~projected/kube-api-access-w9fdf:{mountpoint:/var/lib/kubelet/pods/c4944e38-f0de-40f8-88f3-26fc7b9c2c63/volumes/kubernetes.io~projected/kube-api-access-w9fdf major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f92791-c615-4d6e-88e8-1a3f275aa32a/volumes/kubernetes.io~projected/kube-api-access-hhmtd:{mountpoint:/var/lib/kubelet/pods/c6f92791-c615-4d6e-88e8-1a3f275aa32a/volumes/kubernetes.io~projected/kube-api-access-hhmtd major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6f92791-c615-4d6e-88e8-1a3f275aa32a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/c6f92791-c615-4d6e-88e8-1a3f275aa32a/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7639481-1e64-49cc-a292-5cfa0cf76416/volumes/kubernetes.io~projected/kube-api-access-l7bmn:{mountpoint:/var/lib/kubelet/pods/d7639481-1e64-49cc-a292-5cfa0cf76416/volumes/kubernetes.io~projected/kube-api-access-l7bmn major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/daacb5ca-b3fa-4344-add9-c451fc1e5e25/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/daacb5ca-b3fa-4344-add9-c451fc1e5e25/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1321427-04cc-401b-8df9-43b3664b0720/volumes/kubernetes.io~projected/kube-api-access-z2s4p:{mountpoint:/var/lib/kubelet/pods/e1321427-04cc-401b-8df9-43b3664b0720/volumes/kubernetes.io~projected/kube-api-access-z2s4p major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e487a2e3-2c52-4dea-80a6-fdb09e9229ae/volumes/kubernetes.io~projected/kube-api-access-dz48t:{mountpoint:/var/lib/kubelet/pods/e487a2e3-2c52-4dea-80a6-fdb09e9229ae/volumes/kubernetes.io~projected/kube-api-access-dz48t major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebc19999-3d20-4929-93a6-cfa85a9074d3/volumes/kubernetes.io~projected/kube-api-access-ln2c5:{mountpoint:/var/lib/kubelet/pods/ebc19999-3d20-4929-93a6-cfa85a9074d3/volumes/kubernetes.io~projected/kube-api-access-ln2c5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed959a30-6e00-49b5-aceb-c543a68b9d3e/volumes/kubernetes.io~projected/kube-api-access-ljjhb:{mountpoint:/var/lib/kubelet/pods/ed959a30-6e00-49b5-aceb-c543a68b9d3e/volumes/kubernetes.io~projected/kube-api-access-ljjhb major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~projected/kube-api-access-dvfsh:{mountpoint:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~projected/kube-api-access-dvfsh major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/401a7951a5aa9a2c4f2208ed3eb0d3760f59b9709e655e8048b87243b288952d/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/000f103bc989e079adbfe0fff0c0e5ff6c9ceb1e4f4dbe9bc5a5369bdb9ef6d3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7524baa084efc09436e4c6e4c4f32a3415a115a19506f16396156064c3c3ddda/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/589ec75b0394caf3fb25b95ccfa1b9614aa5f5be3cef7d6c9c8137a4b96c4c5f/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/32b7843960a307522795a516ee7e91d9fcb801f276125bd51d541b6082fdadac/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/3f7ec1e57a98ba333c05336cdabcf37053bd1aaa606723e979d6a4d66b9bb28c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fe537d76fbc6030422a79561b937880708c1ea5f728fee4eff162c4a01826d4c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4b200252f8ed376c7e1a8dbfc779539275ba9345fd59aa03efedffef5f5a7168/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/519c2a382d081f6cedcae86863fae50d2df237d58a349f3feee2c8a2f1fce58e/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-131:{mountpoint:/var/lib/containers/storage/overlay/5ae6c08eed046490805c89b3db6801ca582764adec25096628dae94458506f82/merged major:0 minor:131 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/60767706876bef5415ee130740a8e50a16c73c1b249b04134ec58741af616e8c/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/fc457a1545fd0c04ce1b8f3fc97bacf1b9e5d78d66a5c81f986e563d97504a9c/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1b9e4ed90818eab2223b3d47442c0dfbc4172d309fee944462e78b50acd57423/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ab12a3eb54e7fae4b8d281d22f7e5342ab9a849b65c1c091dd770a5d3414ff98/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ceafd648e0de8d94f26318f19b9843358d6a847090abcacd0142b6055e20733b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bf54653c506b6d8539ae3cbabb8a88edc65db0df2be2882e0fbb298a1fb082e3/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/24ba4d52b2f00e4335f4a191ea6087e3dfa9b13dc9387a2ac2b60a84b47fdd32/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/b4df80da4a0622f56626d68062369b7248f130261611e5b23ba5499fe6c2d6aa/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7413f506088220c844ca4a05f15a81e1732ae5848eb66620904cd14ff855fd30/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/934bb208a657f1a5915b787dac0740bf6371312c6dd916251558dc9a770e4a79/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f12e106aaf1d48dac5753825385fca1b93a48fc763079bce70eb4c72501ef664/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/747d56b23d47936306fb4bb01cb81ccc7a4399a498cc5217e2624545c3dfc692/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/9ad757299847aeb3fef60c66b98a9caa57a26735f99c2a8dc9b83c5dde9a1d0f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/17fd23c6bb8dca192e29b56e663fb2f3db58e2dee601b754d25c10e9492a8aa0/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/395f340169be2d932175311f7b4720072eb8b7ea41585e94b589ef8e4b51d4ad/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/de7373f244f51c1300f5860291dc5093b78f8ccf666ec311aaad144209aa3772/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/a6f10e7e9ef8acfaa60b598820524eb55471c39873592bafc57925d74385e402/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/2ecc6426a90b0fe540a3a64b71ec0831c6b63c97a2486a06dcf24a743aa10632/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/62b214c5142be676e517579a5bdf211a4bbb02850b532add6879c1adda3d8644/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/cfc81ade10a1d14a8b85da9f14924789f00a59e0795e5eb3a589501b0867dea9/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/5ba283f9326a1aeaee260b5385b751a7b70b375b0e61b75ef0799b2cc0e4a620/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/d80ad25849bfee592edc476742663922d59bf7b384c6d5a9fe147007bb22a739/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/a42f2dc27fdbdfffb3815a5df4035642844d9d787e6b7b2bded4ce1ec7fa19d0/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/ae07045880d22d331f41752063d4b081d6895b101cbfc45e48297419f9734e8e/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/7edacea38e56910a64af080e74cb2338122c2e999a985209d60a0c0d24a182f3/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/74e10b6361163ed3ec5c6e119315daef4c86db3d7f2dfead8c5170cd237c0480/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/eaaf8528b27a4134d6a49a823d611bdf8b08d7d4fec85549e35c167357ae1d68/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/897e8689fafc2ffb4032895bb5827e294c17920bab44d6e2305d67408df393c2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/dcfef80afc2afef8063a3d6bdee9b9bbc0a8aed5c1d9fed3f895df838635cbbc/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/a687dfdc3994365a266d13455ad9ce060ceebf2577c44728cbf25b423fea75a9/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/fac0da8a218c86625998bf215a62d2098e3fc1b7e5073173f26be80d51928942/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/56c383a01c987f3481ba1b75b9af4a7621335c2ac1f5b9809b8164e48ea0e4ef/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/764a96a096c8ac1bb98f3f2b9d78de642aed4f3f543b7da2e002ce745b10ac42/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/b9f0832c19aafb3151074214abb2554298995bb1418dfd37b199782a8c97c3e6/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/7816dc3907a81379880bcaac2d870ea1e13b5d795ef6806affde2681e8d67be6/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/acf65914762a9c787701f2e099d57b5c1d18678fd5bbd35ae5a704398f69fc3b/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/063979cb191b90a699824ba87ddad1f5463fd6cd9fe9358f59deff4b2599b563/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/207ac2fe8f83a00100ee27c996a6d3119341517dcbc436e24034dab276aba14f/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/8d49c936bcf630e923c2a6b672199b326d60a097f7e44ab2ba3ac9565c2a2691/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/60140f8be4f6b10a8cc1ce394516c992ec84bd8add9ddceb592db30524f737f5/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c67b0b2a5508ef3cdcf8dfc190de81cfaf3359405345e41da9287c5ab23d322f/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/32580f47ae36b694046b1ae94d60bda2738bbc20a1fe8e15c6fae76a56814ed8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/9d519ffc6dfd668ada04f0f848da6fa719f95581888bf92e3b675e983fc4c293/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/51569f94bcfd413d5a60dc4971caa628bedeb8e8b7ed3b29fc876fc8af6c317a/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/bf17576bdd58aeeb18d9c1c944ada23397c2caad434ea6b27df2b132f51e5d38/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c1124d807566832a2fdc85336addf7677a7f39b0c7e7bbd0add56d81b6b2de4a/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/1db7da408dd75f4ac91878748ce3df57b6054984d0c985e113ccb5b6ea916316/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/717f4150f79c30dfc48c43533ff410089fa97dd55ce5058efad5c17a7efed585/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c3f2464667d4cf40991c15e4e5a55a48996ddf856ece97fcc0695b6782924b49/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7b44391a4f99ea00950bb805ebf6ee8435f215b6574b5cf5db0e9373eae8e099/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.992439    5025 manager.go:217] Machine: {Timestamp:2026-10-30 18:36:57.991853425 +0000 UTC m=+0.072439623 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100052 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a2c0f6017d1ba85cb50e341645233 SystemUUID:ec2a2c0f-6017-d1ba-85cb-50e341645233 BootID:19716777-e17c-43e3-984f-f02f1117aa59 Filesystems:[{Device:/var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f92791-c615-4d6e-88e8-1a3f275aa32a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb/volumes/kubernetes.io~projected/kube-api-access-m7pwv DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e8d409944524c9be12242c109ff945511d8360e905cf992cd39d9d2f68bb1a4/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1/userdata/shm DeviceMajor:0 DeviceMinor:335 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0/volumes/kubernetes.io~projected/kube-api-access-r6fmf DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b9790e2f-d93d-47f4-aa83-6c8a54567513/volumes/kubernetes.io~projected/kube-api-access-8dl7q DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8e96c52d-079e-4b90-b722-b69ad8b12730/volumes/kubernetes.io~projected/kube-api-access-k7ttf DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a705f33-1517-4f85-8eca-10395ff58b12/volumes/kubernetes.io~projected/kube-api-access-r248d DeviceMajor:0 DeviceMinor:361 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e1321427-04cc-401b-8df9-43b3664b0720/volumes/kubernetes.io~projected/kube-api-access-z2s4p DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d7639481-1e64-49cc-a292-5cfa0cf76416/volumes/kubernetes.io~projected/kube-api-access-l7bmn DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e78e1e0874181c9ac5126599740a232797b7e2b873614e291dc812a53cc9474d/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ebc19999-3d20-4929-93a6-cfa85a9074d3/volumes/kubernetes.io~projected/kube-api-access-ln2c5 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e63e724d48f0f19b7214cc23c6af655ca1f287b1ee37570f91201c4145de257d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e292958ee020bc49993fbd96dc254942c8ac03afeaf637d1c24b6989d9d6582d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a56de33d-a1ed-44e6-863f-11ae45cec41f/volumes/kubernetes.io~projected/kube-api-access-95vb9 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d1666c09505424df2d98850264bb43c073b50ea18afc3b310c59d853c111ee0/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/511cf18c-8d23-40e7-9e68-f683e3fe9e7d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6f0cdf2a-c119-4ffa-8ae0-395f249cf285/volumes/kubernetes.io~projected/kube-api-access-tc7ht DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a159864-3010-4195-84de-1407b16eb5c2/volumes/kubernetes.io~projected/kube-api-access-pjnpk DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5e89546-8707-4c00-9bf2-826c6fbcd761/volumes/kubernetes.io~projected/kube-api-access-k58pb DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80b240bd-b789-4575-8a22-f2773e4ef78d/volumes/kubernetes.io~projected/kube-api-access-2mprt DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/336c71cc-488f-44cc-8d22-5c8bac5f0bda/volumes/kubernetes.io~projected/kube-api-access-nnflj DeviceMajor:0 DeviceMinor:339 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a705f33-1517-4f85-8eca-10395ff58b12/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:360 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/acca8ca1-d447-4a1f-bd99-8dc48014e033/volumes/kubernetes.io~projected/kube-api-access-5x95b DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d51c59f-38ab-4311-bf40-cf79edf28d4a/volumes/kubernetes.io~projected/kube-api-access-9d82m DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e6a72c5-3683-4fc8-8525-e82baed6a4ad/volumes/kubernetes.io~projected/kube-api-access-46xgz DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3c70ee516a89708a242cefbbf143a08d3e24beb903c1b1b14245c0441e1c7f3/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d79841a1a769d7f8cdd3485314cedb0af23031618e37da1c66bb9d8b2391c2d/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80b240bd-b789-4575-8a22-f2773e4ef78d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a32792a45e9c049059e85402f62eca5f3a15667728952dc5c1461d279f6faddf/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d59ccdf-e915-4825-9055-c6b9f4c1c76d/volumes/kubernetes.io~projected/kube-api-access-zqctd DeviceMajor:0 DeviceMinor:338 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-131 DeviceMajor:0 DeviceMinor:131 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c6f92791-c615-4d6e-88e8-1a3f275aa32a/volumes/kubernetes.io~projected/kube-api-access-hhmtd DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e/volumes/kubernetes.io~projected/kube-api-access-mhrlf DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/daacb5ca-b3fa-4344-add9-c451fc1e5e25/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:278 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e0626d39502c3fc05873906b92be1a6ef36223010e13cdfa69152ae3e504482/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c33be6c60ef24e58a631561748f80157c89f51424eec89fdd51e5d9eec2a1a2e/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed959a30-6e00-49b5-aceb-c543a68b9d3e/volumes/kubernetes.io~projected/kube-api-access-ljjhb DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b84c7fdfd0ff925d3a6890b81d3ec70a4aaca3768a37e4ef9839e0b4475b0608/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~projected/kube-api-access-ddgrz DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/77bb5229-a381-4bf0-82ee-e4eb959cf7eb/volumes/kubernetes.io~projected/kube-api-access-n59hl DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e0ff618735547c8ea5cf78e9c01829e04b0d9871061958c101fd654355b8203/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/139df7c01e09b92f6e4b7e1f66c1994088ad5cb1f9cfcd9a16b7f58b6cab217b/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e2e5b162bbebecc4eb3a01639fab0ccf23d39a84a4c22bd72d4b5cc09ea4349/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b20c7ba1b9740fe09f9f44e8def6991f7961412588fbb6dbeb1e517d8027b0e/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/670b5b50-4c64-404e-b62c-9fe151aa58b2/volumes/kubernetes.io~projected/kube-api-access-r776j DeviceMajor:0 DeviceMinor:323 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bfc21e7a96ed52386bc582f793ccf51e5ef05af1a33e491a60544c19be825b8c/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e96c52d-079e-4b90-b722-b69ad8b12730/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/accf98f1-25fe-4fb4-b63a-05ad93a65d1f/volumes/kubernetes.io~projected/kube-api-access-mh99h DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9790e2f-d93d-47f4-aa83-6c8a54567513/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6853f8fb0b949d504409e0e1590c637219deedf5a0b8070bb21d423e02cbcbcc/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/90552f94-1a43-466e-9285-b9e00c4ed560/volumes/kubernetes.io~projected/kube-api-access-4njk2 DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e47c8cbc0af923b5cde2c263466bb43fd9b4f23e12ead943f5c8670666440005/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/volumes/kubernetes.io~projected/kube-api-access-c8jrs DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e487a2e3-2c52-4dea-80a6-fdb09e9229ae/volumes/kubernetes.io~projected/kube-api-access-dz48t DeviceMajor:0 DeviceMinor:220 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes/kubernetes.io~projected/kube-api-access-dvfsh DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ac122194e6e5df57c31b21a993a4ec0409a7bbafd0c1ffc6143d01e9d00d789/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes/kubernetes.io~projected/kube-api-access-5xv4l DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4944e38-f0de-40f8-88f3-26fc7b9c2c63/volumes/kubernetes.io~projected/kube-api-access-w9fdf DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e1f7fdbbd45cae2de375c305a08e7c7ca42fb3dba477820d7647c62af9b8463/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6596f80cb7c7bd1c0817d5bb20313f5345d7aeefb0e4a79548ad3b46489f2327/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f44bc15e73a6cf766f33ee46d9852e77ee75c82999b518ef9c720bc3b1943f4/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/271d8b24c9d5a4231742055df0db674e938ba0d77a8ac519038722a9c8e2ee1d/userdata/shm DeviceMajor:0 DeviceMinor:344 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19d1dc0f-27db-4784-916c-dbf422a4d602/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:139df7c01e09b92 MacAddress:aa:fa:eb:d6:82:fb Speed:10000 Mtu:8901} {Name:1b20c7ba1b9740f MacAddress:a2:bb:0b:c3:50:50 Speed:10000 Mtu:8901} {Name:271d8b24c9d5a42 MacAddress:aa:c9:78:43:f6:e7 Speed:10000 Mtu:8901} {Name:2d1666c09505424 MacAddress:5a:d2:5b:34:66:4f Speed:10000 Mtu:8901} {Name:41d8c76cd0a70c3 MacAddress:86:4e:80:f7:01:02 Speed:10000 Mtu:8901} {Name:4e1f7fdbbd45cae MacAddress:02:fa:49:36:d4:63 Speed:10000 Mtu:8901} {Name:5e2e5b162bbebec MacAddress:5a:af:89:1f:5c:15 Speed:10000 Mtu:8901} {Name:5e8d409944524c9 MacAddress:f6:60:41:65:85:f6 Speed:10000 Mtu:8901} {Name:6596f80cb7c7bd1 MacAddress:de:c5:0d:57:af:e1 Speed:10000 Mtu:8901} {Name:6853f8fb0b949d5 MacAddress:06:28:a6:37:c0:06 Speed:10000 Mtu:8901} {Name:7f44bc15e73a6cf MacAddress:ce:db:9c:8f:e6:14 Speed:10000 Mtu:8901} {Name:a32792a45e9c049 MacAddress:1a:e7:99:76:d9:fd Speed:10000 Mtu:8901} {Name:b84c7fdfd0ff925 MacAddress:5a:d5:16:d3:db:21 Speed:10000 Mtu:8901} {Name:bfc21e7a96ed523 MacAddress:92:68:26:d7:90:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:df:b0:b4:23 Speed:0 Mtu:9001} {Name:br-int MacAddress:16:25:12:18:cc:e2 Speed:0 Mtu:8901} {Name:c33be6c60ef24e5 MacAddress:2e:a4:f1:4b:3c:14 Speed:10000 Mtu:8901} {Name:e78e1e0874181c9 MacAddress:76:79:05:e8:69:7f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:df:b0:b4:23 Speed:0 Mtu:9001} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:4a:8d:6c:d1:46:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.992894    5025 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.992948    5025 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993231    5025 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993257    5025 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-88.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993424    5025 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993433    5025 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993447    5025 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993467    5025 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993487    5025 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993662    5025 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993792    5025 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993846    5025 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993861    5025 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993876    5025 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993893    5025 kubelet.go:405] "Adding apiserver pod source"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.993902    5025 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.994773    5025 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.994787    5025 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:57.998708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:57.996693    5025 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.010510    5025 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.010531    5025 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.010690    5025 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.010983    5025 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011025    5025 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011153    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011168    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011179    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011188    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011198    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011214    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011223    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011233    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011244    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011254    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011269    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011282    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011315    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 18:36:58.012036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.011359    5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 18:36:58.015988 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.015898    5025 server.go:1263] "Started kubelet"
Oct 30 18:36:58.016113 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.016029    5025 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 18:36:58.016188 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.016166    5025 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 18:36:58.016516 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.016496    5025 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 18:36:58.016601 ip-10-0-129-88 systemd[1]: Started Kubernetes Kubelet.
Oct 30 18:36:58.017256 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.016878    5025 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 18:36:58.018312 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.018295    5025 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 18:36:58.019859 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.019836    5025 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 18:36:58.019954 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.019868    5025 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 18:36:58.020023 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.019956    5025 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 18:36:58.020023 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.019964    5025 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 18:36:58.020023 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.020004    5025 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 18:31:17 +0000 UTC" deadline="2028-07-21 19:54:39.14092953 +0000 UTC"
Oct 30 18:36:58.020171 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.020029    5025 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15121h17m41.120902433s"
Oct 30 18:36:58.020171 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.020056    5025 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 18:36:58.023019 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.022996    5025 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023372    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b240bd-b789-4575-8a22-f2773e4ef78d" volumeName="kubernetes.io/secret/80b240bd-b789-4575-8a22-f2773e4ef78d-cert" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023412    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d964528-47c1-4c62-8185-a27cc4cb8447" volumeName="kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023427    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9790e2f-d93d-47f4-aa83-6c8a54567513" volumeName="kubernetes.io/secret/b9790e2f-d93d-47f4-aa83-6c8a54567513-serving-cert" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023440    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f92791-c615-4d6e-88e8-1a3f275aa32a" volumeName="kubernetes.io/secret/c6f92791-c615-4d6e-88e8-1a3f275aa32a-serving-cert" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023454    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="daacb5ca-b3fa-4344-add9-c451fc1e5e25" volumeName="kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023465    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e487a2e3-2c52-4dea-80a6-fdb09e9229ae" volumeName="kubernetes.io/configmap/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-telemetry-config" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023477    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc19999-3d20-4929-93a6-cfa85a9074d3" volumeName="kubernetes.io/projected/ebc19999-3d20-4929-93a6-cfa85a9074d3-kube-api-access-ln2c5" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023488    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d964528-47c1-4c62-8185-a27cc4cb8447" volumeName="kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023502    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d964528-47c1-4c62-8185-a27cc4cb8447" volumeName="kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023515    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e96c52d-079e-4b90-b722-b69ad8b12730" volumeName="kubernetes.io/secret/8e96c52d-079e-4b90-b722-b69ad8b12730-serving-cert" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023529    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="336c71cc-488f-44cc-8d22-5c8bac5f0bda" volumeName="kubernetes.io/projected/336c71cc-488f-44cc-8d22-5c8bac5f0bda-kube-api-access-nnflj" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023541    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e96c52d-079e-4b90-b722-b69ad8b12730" volumeName="kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023552    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e96c52d-079e-4b90-b722-b69ad8b12730" volumeName="kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-snapshots" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023564    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90552f94-1a43-466e-9285-b9e00c4ed560" volumeName="kubernetes.io/empty-dir/90552f94-1a43-466e-9285-b9e00c4ed560-data-volume" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023578    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="accf98f1-25fe-4fb4-b63a-05ad93a65d1f" volumeName="kubernetes.io/empty-dir/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023595    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5e89546-8707-4c00-9bf2-826c6fbcd761" volumeName="kubernetes.io/projected/b5e89546-8707-4c00-9bf2-826c6fbcd761-kube-api-access-k58pb" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023609    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" volumeName="kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023621    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" volumeName="kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023635    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e96c52d-079e-4b90-b722-b69ad8b12730" volumeName="kubernetes.io/projected/8e96c52d-079e-4b90-b722-b69ad8b12730-kube-api-access-k7ttf" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023648    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5e89546-8707-4c00-9bf2-826c6fbcd761" volumeName="kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-env-overrides" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023660    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f92791-c615-4d6e-88e8-1a3f275aa32a" volumeName="kubernetes.io/projected/c6f92791-c615-4d6e-88e8-1a3f275aa32a-kube-api-access-hhmtd" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023672    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1321427-04cc-401b-8df9-43b3664b0720" volumeName="kubernetes.io/projected/e1321427-04cc-401b-8df9-43b3664b0720-kube-api-access-z2s4p" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023693    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" volumeName="kubernetes.io/projected/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-kube-api-access-n59hl" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023707    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d59ccdf-e915-4825-9055-c6b9f4c1c76d" volumeName="kubernetes.io/projected/7d59ccdf-e915-4825-9055-c6b9f4c1c76d-kube-api-access-zqctd" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023719    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d964528-47c1-4c62-8185-a27cc4cb8447" volumeName="kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023731    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d964528-47c1-4c62-8185-a27cc4cb8447" volumeName="kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023743    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9790e2f-d93d-47f4-aa83-6c8a54567513" volumeName="kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023761    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed959a30-6e00-49b5-aceb-c543a68b9d3e" volumeName="kubernetes.io/projected/ed959a30-6e00-49b5-aceb-c543a68b9d3e-kube-api-access-ljjhb" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023781    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d1dc0f-27db-4784-916c-dbf422a4d602" volumeName="kubernetes.io/projected/19d1dc0f-27db-4784-916c-dbf422a4d602-kube-api-access-ddgrz" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023794    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e6a72c5-3683-4fc8-8525-e82baed6a4ad" volumeName="kubernetes.io/projected/1e6a72c5-3683-4fc8-8525-e82baed6a4ad-kube-api-access-46xgz" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023805    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a159864-3010-4195-84de-1407b16eb5c2" volumeName="kubernetes.io/projected/4a159864-3010-4195-84de-1407b16eb5c2-kube-api-access-pjnpk" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023816    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="670b5b50-4c64-404e-b62c-9fe151aa58b2" volumeName="kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023827    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acca8ca1-d447-4a1f-bd99-8dc48014e033" volumeName="kubernetes.io/projected/acca8ca1-d447-4a1f-bd99-8dc48014e033-kube-api-access-5x95b" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023839    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e487a2e3-2c52-4dea-80a6-fdb09e9229ae" volumeName="kubernetes.io/projected/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-kube-api-access-dz48t" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023853    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" volumeName="kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023864    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41314e0f-aadd-4cdf-aac1-93fb90cb9a5f" volumeName="kubernetes.io/projected/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-kube-api-access-c8jrs" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023877    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e96c52d-079e-4b90-b722-b69ad8b12730" volumeName="kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-tmp" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023888    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90552f94-1a43-466e-9285-b9e00c4ed560" volumeName="kubernetes.io/projected/90552f94-1a43-466e-9285-b9e00c4ed560-kube-api-access-4njk2" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023901    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="acca8ca1-d447-4a1f-bd99-8dc48014e033" volumeName="kubernetes.io/configmap/acca8ca1-d447-4a1f-bd99-8dc48014e033-iptables-alerter-script" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023913    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d1dc0f-27db-4784-916c-dbf422a4d602" volumeName="kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-tmp" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023925    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19d1dc0f-27db-4784-916c-dbf422a4d602" volumeName="kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-tuned" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023937    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="334cba69-4d11-4125-9763-d79d806afbc8" volumeName="kubernetes.io/configmap/334cba69-4d11-4125-9763-d79d806afbc8-nginx-conf" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023948    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0cdf2a-c119-4ffa-8ae0-395f249cf285" volumeName="kubernetes.io/projected/6f0cdf2a-c119-4ffa-8ae0-395f249cf285-kube-api-access-tc7ht" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.023984    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e96c52d-079e-4b90-b722-b69ad8b12730" volumeName="kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024000    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7639481-1e64-49cc-a292-5cfa0cf76416" volumeName="kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024013    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebc19999-3d20-4929-93a6-cfa85a9074d3" volumeName="kubernetes.io/empty-dir/ebc19999-3d20-4929-93a6-cfa85a9074d3-tmp-dir" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024025    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" volumeName="kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024037    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511cf18c-8d23-40e7-9e68-f683e3fe9e7d" volumeName="kubernetes.io/secret/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-agent-certs" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024050    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d964528-47c1-4c62-8185-a27cc4cb8447" volumeName="kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024063    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c90acf4-d9a8-4478-a97d-cc5bfe500fa0" volumeName="kubernetes.io/configmap/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-serviceca" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024074    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9790e2f-d93d-47f4-aa83-6c8a54567513" volumeName="kubernetes.io/projected/b9790e2f-d93d-47f4-aa83-6c8a54567513-kube-api-access-8dl7q" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024086    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4944e38-f0de-40f8-88f3-26fc7b9c2c63" volumeName="kubernetes.io/projected/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-kube-api-access-w9fdf" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024098    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6f92791-c615-4d6e-88e8-1a3f275aa32a" volumeName="kubernetes.io/configmap/c6f92791-c615-4d6e-88e8-1a3f275aa32a-config" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024111    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" volumeName="kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024123    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a705f33-1517-4f85-8eca-10395ff58b12" volumeName="kubernetes.io/secret/1a705f33-1517-4f85-8eca-10395ff58b12-signing-key" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024135    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ee97fd8-b164-40ca-a9ff-e53e0f461d9e" volumeName="kubernetes.io/configmap/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-config" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024183    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56de33d-a1ed-44e6-863f-11ae45cec41f" volumeName="kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-cni-binary-copy" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024199    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56de33d-a1ed-44e6-863f-11ae45cec41f" volumeName="kubernetes.io/projected/a56de33d-a1ed-44e6-863f-11ae45cec41f-kube-api-access-95vb9" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024216    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" volumeName="kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024229    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41314e0f-aadd-4cdf-aac1-93fb90cb9a5f" volumeName="kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-default-certificate" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024244    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="511cf18c-8d23-40e7-9e68-f683e3fe9e7d" volumeName="kubernetes.io/configmap/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-konnectivity-ca" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024259    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="670b5b50-4c64-404e-b62c-9fe151aa58b2" volumeName="kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024271    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb" volumeName="kubernetes.io/projected/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb-kube-api-access-m7pwv" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024296    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a705f33-1517-4f85-8eca-10395ff58b12" volumeName="kubernetes.io/configmap/1a705f33-1517-4f85-8eca-10395ff58b12-signing-cabundle" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024312    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41314e0f-aadd-4cdf-aac1-93fb90cb9a5f" volumeName="kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-stats-auth" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024346    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="670b5b50-4c64-404e-b62c-9fe151aa58b2" volumeName="kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024360    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" volumeName="kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024373    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d51c59f-38ab-4311-bf40-cf79edf28d4a" volumeName="kubernetes.io/projected/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kube-api-access-9d82m" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024386    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="accf98f1-25fe-4fb4-b63a-05ad93a65d1f" volumeName="kubernetes.io/configmap/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-config-volume" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024399    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5e89546-8707-4c00-9bf2-826c6fbcd761" volumeName="kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-config" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024413    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" volumeName="kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024426    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a56de33d-a1ed-44e6-863f-11ae45cec41f" volumeName="kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-daemon-config" seLinuxMountContext=""
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024431    5025 factory.go:55] Registering systemd factory
Oct 30 18:36:58.024452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024454    5025 factory.go:223] Registration of the systemd container factory successfully
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024437    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="accf98f1-25fe-4fb4-b63a-05ad93a65d1f" volumeName="kubernetes.io/projected/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-kube-api-access-mh99h" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024548    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a705f33-1517-4f85-8eca-10395ff58b12" volumeName="kubernetes.io/projected/1a705f33-1517-4f85-8eca-10395ff58b12-kube-api-access-r248d" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024586    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ee97fd8-b164-40ca-a9ff-e53e0f461d9e" volumeName="kubernetes.io/projected/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-kube-api-access-mhrlf" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024611    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="670b5b50-4c64-404e-b62c-9fe151aa58b2" volumeName="kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024626    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" volumeName="kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024639    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" volumeName="kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024652    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80b240bd-b789-4575-8a22-f2773e4ef78d" volumeName="kubernetes.io/projected/80b240bd-b789-4575-8a22-f2773e4ef78d-kube-api-access-2mprt" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024665    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d964528-47c1-4c62-8185-a27cc4cb8447" volumeName="kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024677    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c90acf4-d9a8-4478-a97d-cc5bfe500fa0" volumeName="kubernetes.io/projected/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-kube-api-access-r6fmf" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024689    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5e89546-8707-4c00-9bf2-826c6fbcd761" volumeName="kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024700    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5e89546-8707-4c00-9bf2-826c6fbcd761" volumeName="kubernetes.io/secret/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024712    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9790e2f-d93d-47f4-aa83-6c8a54567513" volumeName="kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-config" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024724    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4944e38-f0de-40f8-88f3-26fc7b9c2c63" volumeName="kubernetes.io/configmap/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-config" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024737    5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ee97fd8-b164-40ca-a9ff-e53e0f461d9e" volumeName="kubernetes.io/secret/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-serving-cert" seLinuxMountContext=""
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024749    5025 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024755    5025 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024763    5025 factory.go:153] Registering CRI-O factory
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024779    5025 factory.go:223] Registration of the crio container factory successfully
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024830    5025 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024848    5025 factory.go:103] Registering Raw factory
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.024862    5025 manager.go:1199] Started watching for new ooms in manager
Oct 30 18:36:58.027616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.025199    5025 manager.go:319] Starting recovery of all containers
Oct 30 18:36:58.041944 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.041567    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/1.log"
Oct 30 18:36:58.042797 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.042772    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/1.log"
Oct 30 18:36:58.048304 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.048258    5025 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 18:36:58.049889 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.049561    5025 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 18:36:58.049889 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.049589    5025 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 18:36:58.049889 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.049607    5025 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 18:36:58.049889 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.049648    5025 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 18:36:58.095792 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.095761    5025 generic.go:356] "Generic (PLEG): container finished" podID="7be7e0652d2c7a6c3979448764fb16b6" containerID="158398e0acf3fb176ba22f7d6b93572660a1088c1c59ff77803d35f251381ce3" exitCode=0
Oct 30 18:36:58.099864 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.099841    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/1.log"
Oct 30 18:36:58.099998 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.099872    5025 generic.go:356] "Generic (PLEG): container finished" podID="b9790e2f-d93d-47f4-aa83-6c8a54567513" containerID="826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870" exitCode=255
Oct 30 18:36:58.123543 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.123515    5025 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:58.124504 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.124473    5025 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="7a38b0370b47fc210609aa7b088de80e74045c4afed00f71a555101464b434c0" exitCode=0
Oct 30 18:36:58.124622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.124503    5025 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="e30d10b2a7559143fae6fbf6de48715885b63931b69c7d1a5fe9903e6e2259c6" exitCode=0
Oct 30 18:36:58.124622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.124517    5025 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="218db7ddac86120c26c1a3300cefa56dc37aa32f17217ca7e6bc663f3590e567" exitCode=0
Oct 30 18:36:58.124622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.124530    5025 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="2dbfce9b0f76a54fcae83e59fe5af47eb83b0c6fc5c58b734554f55ad2188ef2" exitCode=0
Oct 30 18:36:58.124622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.124535    5025 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="29877a2ab78d5c27f3374e88435da416203f51d6c97f8cc3a73a7a45a6682427" exitCode=0
Oct 30 18:36:58.124622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.124540    5025 generic.go:356] "Generic (PLEG): container finished" podID="77bb5229-a381-4bf0-82ee-e4eb959cf7eb" containerID="da595da6965f804b3bb01ac9e2ebaf716e55737f8dd8cac12fcb41ea636469ff" exitCode=0
Oct 30 18:36:58.138148 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.138132    5025 manager.go:324] Recovery completed
Oct 30 18:36:58.150412 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.150381    5025 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 18:36:58.150576 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.150558    5025 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:58.162490 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.162473    5025 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:58.164562 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.164541    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:58.164667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.164583    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:58.164667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.164599    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:58.168002 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.167986    5025 cpu_manager.go:226] "Starting" policy="none"
Oct 30 18:36:58.168002 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168000    5025 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 18:36:58.168002 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168003    5025 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:58.168183 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168022    5025 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 18:36:58.168183 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168176    5025 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 18:36:58.168309 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168193    5025 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 18:36:58.168309 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168235    5025 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 18:36:58.168309 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168244    5025 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 18:36:58.168309 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168252    5025 policy_none.go:50] "Start"
Oct 30 18:36:58.168309 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168272    5025 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 18:36:58.168309 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.168287    5025 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 18:36:58.169461 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.169441    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:58.169551 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.169477    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:58.169551 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.169490    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:58.184210 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.184192    5025 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 18:36:58.184210 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.184212    5025 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 18:36:58.184341 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.184220    5025 policy_none.go:44] "Start"
Oct 30 18:36:58.188607 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.188582    5025 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 18:36:58.192053 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192040    5025 manager.go:341] "Starting Device Plugin manager"
Oct 30 18:36:58.192137 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.192064    5025 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 18:36:58.192137 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192072    5025 server.go:87] "Starting device plugin registration server"
Oct 30 18:36:58.192410 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192390    5025 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 18:36:58.192459 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192402    5025 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 18:36:58.192515 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192478    5025 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 18:36:58.192652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192611    5025 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 18:36:58.192711 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192650    5025 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 18:36:58.192711 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192669    5025 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 18:36:58.193518 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.192721    5025 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:36:58.19264075Z","UUID":"61efea14-95a5-486d-8c22-9e83631e7064","Handler":null,"Name":"","Endpoint":""}
Oct 30 18:36:58.193849 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.193822    5025 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 18:36:58.194083 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.194064    5025 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 18:36:58.194175 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.194098    5025 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-88.us-west-2.compute.internal\" not found"
Oct 30 18:36:58.195167 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.195148    5025 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 18:36:58.195262 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.195183    5025 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 18:36:58.199223 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.199206    5025 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:58.199316 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.199286    5025 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:58.246520 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.246491    5025 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 18:36:58.293371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.293340    5025 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 18:36:58.294457 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.294438    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 18:36:58.294547 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.294470    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 18:36:58.294547 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.294483    5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-88.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 18:36:58.294547 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.294525    5025 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.351154 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.351110    5025 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"]
Oct 30 18:36:58.351873 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.351825    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"7be7e0652d2c7a6c3979448764fb16b6","Type":"ContainerStarted","Data":"60bdc0bd3cec7891bd1e944d2d86d7253931c4e94d9a1c126fdda7f45bafb9c2"}
Oct 30 18:36:58.351979 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.351887    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"7be7e0652d2c7a6c3979448764fb16b6","Type":"ContainerDied","Data":"158398e0acf3fb176ba22f7d6b93572660a1088c1c59ff77803d35f251381ce3"}
Oct 30 18:36:58.351979 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.351909    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"7be7e0652d2c7a6c3979448764fb16b6","Type":"ContainerStarted","Data":"7726c4eab9e973d569c2c69c7ad8ec25fc318532cd7fe91e310c6ccffbf05797"}
Oct 30 18:36:58.351979 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.351933    5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.351979 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.351966    5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.352126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.351939    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"82007190ff1dd353d8af659976a136ef","Type":"ContainerStarted","Data":"1dbef6ba6555bead4a8dc3aa8e0de1ef45fc9a0e2f92292e54b754cef3940b03"}
Oct 30 18:36:58.352126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.352028    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal" event={"ID":"82007190ff1dd353d8af659976a136ef","Type":"ContainerStarted","Data":"0793291e714bc6bf0d413b8288ccfd34c297f8b8bbb9873ef46091f3111efd32"}
Oct 30 18:36:58.401872 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.401809    5025 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.401872 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.401873    5025 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.422518 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.422496    5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:58.422518 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.422497    5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:58.422663 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.422546    5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.422663 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:58.422574    5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.495210 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.495184    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.495354 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.495218    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.495354 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.495246    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82007190ff1dd353d8af659976a136ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"82007190ff1dd353d8af659976a136ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.595433 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.595402    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.595433 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.595436    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.595616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.595453    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82007190ff1dd353d8af659976a136ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"82007190ff1dd353d8af659976a136ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.595616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.595506    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/82007190ff1dd353d8af659976a136ef-config\") pod \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"82007190ff1dd353d8af659976a136ef\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.595616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.595506    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.595616 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.595528    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7be7e0652d2c7a6c3979448764fb16b6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" (UID: \"7be7e0652d2c7a6c3979448764fb16b6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:58.995665 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:58.995636    5025 apiserver.go:52] "Watching apiserver"
Oct 30 18:36:59.128816 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.128783    5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:59.129112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.129000    5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:59.200105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.200072    5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:59.200265 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.200122    5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:59.200265 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.200144    5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 18:36:59.200265 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.200181    5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 18:36:59.200483 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.200468    5025 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 18:36:59.201189 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.201167    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rw5pb","kube-system/kas-connection-checker-685895c7c9-vvw8v","openshift-image-registry/node-ca-zfktp","openshift-multus/multus-2kfph","openshift-image-registry/image-registry-b895475ff-2t4mb","openshift-insights/periodic-gathering-69wwp-9m4cv","kube-system/kas-connection-checker-685895c7c9-8cw5s","openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s","openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz","openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v","openshift-network-operator/iptables-alerter-fcsph","openshift-ovn-kubernetes/ovnkube-node-5lk89","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67","openshift-service-ca/service-ca-5fffc97bdf-vg6tg","openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr","kube-system/kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal","openshift-ingress-canary/ingress-canary-dhgxq","openshift-network-diagnostics/network-check-target-hktmd","openshift-insights/insights-runtime-extractor-d52ch","openshift-insights/insights-operator-7c4986d449-mx59k","openshift-multus/network-metrics-daemon-fv5vj","openshift-dns/dns-default-gbsgk","openshift-image-registry/image-registry-8d9cdb74b-n2d7w","openshift-cluster-node-tuning-operator/tuned-m8v44","openshift-console-operator/console-operator-7cd56b8dbc-sfn74","openshift-dns/node-resolver-s6975","openshift-network-console/networking-console-plugin-86c478bb88-s4bss","kube-system/konnectivity-agent-5b6w7","kube-system/global-pull-secret-syncer-k8tk8","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m","openshift-multus/multus-additional-cni-plugins-m4dhb","openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-88.us-west-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k","openshift-ingress/router-default-b457d476c-cmkxg"]
Oct 30 18:36:59.205420 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.204587    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.205543 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.205513    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.206583 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.206564    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.207306 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.207291    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.208304 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.208288    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:59.211567 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.211546    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:59.214753 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.214734    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.215419 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.215404    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.217407 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.217393    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.221936 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.221917    5025 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 18:36:59.231028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.230993    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:59.255483 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.255421    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 18:36:59.255943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.255928    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 18:36:59.259726 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.259706    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 18:36:59.259813 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.259772    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 18:36:59.259813 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.259781    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 18:36:59.260089 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260067    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 18:36:59.260176 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260162    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.260176 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260171    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.260287 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260207    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.260458 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260386    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 18:36:59.260458 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260439    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.260586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260495    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.260586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260515    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 18:36:59.260586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260515    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.260755 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260584    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.260755 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260604    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.260755 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260652    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.260755 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260689    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-bk862\""
Oct 30 18:36:59.260938 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260893    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 18:36:59.260997 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260967    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.260997 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.260974    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.261096 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.261009    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 18:36:59.261096 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.261056    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 18:36:59.261171 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.261111    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.299454 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299429    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8jrs\" (UniqueName: \"kubernetes.io/projected/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-kube-api-access-c8jrs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.299572 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299462    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x95b\" (UniqueName: \"kubernetes.io/projected/acca8ca1-d447-4a1f-bd99-8dc48014e033-kube-api-access-5x95b\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:59.299572 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299485    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.299572 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299520    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.299572 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299555    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-node-log\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.299782 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299582    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.299782 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299609    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.299782 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299671    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-cnibin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.299782 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299718    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhrlf\" (UniqueName: \"kubernetes.io/projected/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-kube-api-access-mhrlf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:59.299940 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299767    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-kubelet\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.299940 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299919    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-netd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.300031 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299955    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-modprobe-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.300031 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.299973    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-conf\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.300031 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300015    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-config-volume\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.300171 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300066    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90552f94-1a43-466e-9285-b9e00c4ed560-crio-socket\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.300171 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300107    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9790e2f-d93d-47f4-aa83-6c8a54567513-serving-cert\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.300171 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300158    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-slash\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300200    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-config\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300221    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-trusted-ca\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300238    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dl7q\" (UniqueName: \"kubernetes.io/projected/b9790e2f-d93d-47f4-aa83-6c8a54567513-kube-api-access-8dl7q\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300255    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a705f33-1517-4f85-8eca-10395ff58b12-signing-cabundle\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300269    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/334cba69-4d11-4125-9763-d79d806afbc8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300286    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300295    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9790e2f-d93d-47f4-aa83-6c8a54567513-serving-cert\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.300388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300302    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvfsh\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300418    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-config\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300428    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc19999-3d20-4929-93a6-cfa85a9074d3-tmp-dir\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300462    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46xgz\" (UniqueName: \"kubernetes.io/projected/1e6a72c5-3683-4fc8-8525-e82baed6a4ad-kube-api-access-46xgz\") pod \"kas-connection-checker-685895c7c9-rw5pb\" (UID: \"1e6a72c5-3683-4fc8-8525-e82baed6a4ad\") " pod="kube-system/kas-connection-checker-685895c7c9-rw5pb"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300509    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebc19999-3d20-4929-93a6-cfa85a9074d3-tmp-dir\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300512    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300615    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300636    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-cni-binary-copy\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300662    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.300760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300707    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300776    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4njk2\" (UniqueName: \"kubernetes.io/projected/90552f94-1a43-466e-9285-b9e00c4ed560-kube-api-access-4njk2\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300811    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-conf-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300840    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjhb\" (UniqueName: \"kubernetes.io/projected/ed959a30-6e00-49b5-aceb-c543a68b9d3e-kube-api-access-ljjhb\") pod \"volume-data-source-validator-58c9f5f46d-rcf9k\" (UID: \"ed959a30-6e00-49b5-aceb-c543a68b9d3e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300884    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-netns\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300933    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300942    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysconfig\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.300986    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e96c52d-079e-4b90-b722-b69ad8b12730-serving-cert\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301038    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-tmp-dir\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.301112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301102    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xv4l\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.301372 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301141    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-tmp-dir\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.301372 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301178    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-host\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.301372 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301261    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqctd\" (UniqueName: \"kubernetes.io/projected/7d59ccdf-e915-4825-9055-c6b9f4c1c76d-kube-api-access-zqctd\") pod \"migrator-66df5d496b-8lkzz\" (UID: \"7d59ccdf-e915-4825-9055-c6b9f4c1c76d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"
Oct 30 18:36:59.301372 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301366    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-systemd-units\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.301490 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301385    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-k8s-cni-cncf-io\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.301490 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301404    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:59.301490 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301423    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc7ht\" (UniqueName: \"kubernetes.io/projected/6f0cdf2a-c119-4ffa-8ae0-395f249cf285-kube-api-access-tc7ht\") pod \"network-check-source-6f78d59f99-wf52v\" (UID: \"6f0cdf2a-c119-4ffa-8ae0-395f249cf285\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"
Oct 30 18:36:59.301490 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301453    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-var-lib-kubelet\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.301490 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301478    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-log-socket\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.301661 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301496    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-script-lib\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.301661 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301516    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.301661 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301583    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.301661 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301636    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnflj\" (UniqueName: \"kubernetes.io/projected/336c71cc-488f-44cc-8d22-5c8bac5f0bda-kube-api-access-nnflj\") pod \"migrator-66df5d496b-6lp98\" (UID: \"336c71cc-488f-44cc-8d22-5c8bac5f0bda\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"
Oct 30 18:36:59.301802 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301688    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.301802 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301792    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-var-lib-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.301893 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301806    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.301893 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301835    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-etc-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.301893 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301863    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-tmp\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.301893 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301885    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.302091 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301951    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-tmp\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.302091 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.301978    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-run\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.302091 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302029    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.302091 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302080    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-env-overrides\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.302203 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302151    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-kubernetes\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.302203 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302174    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-multus\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.302270 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302201    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.302270 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302232    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cnibin\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.302361 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302268    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.302361 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302310    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-host\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:59.302440 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302366    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.302440 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302402    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f92791-c615-4d6e-88e8-1a3f275aa32a-config\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:59.302440 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302434    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-bin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.302557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302456    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-multus-certs\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.302557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302471    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-bin\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.302557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302508    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mprt\" (UniqueName: \"kubernetes.io/projected/80b240bd-b789-4575-8a22-f2773e4ef78d-kube-api-access-2mprt\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:59.302557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302529    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-stats-auth\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.302557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302550    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c6f92791-c615-4d6e-88e8-1a3f275aa32a-config\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:59.302730 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302667    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.302730 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302699    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.302730 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302714    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-stats-auth\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.302829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302735    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.302829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302789    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58pb\" (UniqueName: \"kubernetes.io/projected/b5e89546-8707-4c00-9bf2-826c6fbcd761-kube-api-access-k58pb\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.302829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302821    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-daemon-config\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.302907 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302891    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:59.302953 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302912    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r776j\" (UniqueName: \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.302953 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302928    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acca8ca1-d447-4a1f-bd99-8dc48014e033-iptables-alerter-script\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:59.302953 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302946    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-dbus\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.303062 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.302967    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.303062 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303025    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.303062 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303047    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-config\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.303198 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303063    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovn-node-metrics-cert\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.303198 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303078    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-snapshots\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.303198 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303095    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.303198 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303117    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh99h\" (UniqueName: \"kubernetes.io/projected/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-kube-api-access-mh99h\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.303198 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303163    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-hostroot\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.303388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303214    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.303388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303251    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-system-cni-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.303388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303285    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.303388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303314    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b240bd-b789-4575-8a22-f2773e4ef78d-cert\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:59.303388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303359    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-kubelet-config\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.303388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303366    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-snapshots\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.303388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303384    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.303653 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303413    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n59hl\" (UniqueName: \"kubernetes.io/projected/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-kube-api-access-n59hl\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.303653 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303454    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.303653 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303494    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2s4p\" (UniqueName: \"kubernetes.io/projected/e1321427-04cc-401b-8df9-43b3664b0720-kube-api-access-z2s4p\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:59.303653 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303545    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90552f94-1a43-466e-9285-b9e00c4ed560-data-volume\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.303653 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303587    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acca8ca1-d447-4a1f-bd99-8dc48014e033-host-slash\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:59.303653 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303625    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebc19999-3d20-4929-93a6-cfa85a9074d3-hosts-file\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303669    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-tmp\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303745    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/90552f94-1a43-466e-9285-b9e00c4ed560-data-volume\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303752    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e96c52d-079e-4b90-b722-b69ad8b12730-tmp\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303761    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhmtd\" (UniqueName: \"kubernetes.io/projected/c6f92791-c615-4d6e-88e8-1a3f275aa32a-kube-api-access-hhmtd\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303793    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-system-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303822    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-agent-certs\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303849    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjnpk\" (UniqueName: \"kubernetes.io/projected/4a159864-3010-4195-84de-1407b16eb5c2-kube-api-access-pjnpk\") pod \"kas-connection-checker-685895c7c9-8cw5s\" (UID: \"4a159864-3010-4195-84de-1407b16eb5c2\") " pod="kube-system/kas-connection-checker-685895c7c9-8cw5s"
Oct 30 18:36:59.303877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303874    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303916    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-ovn\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303962    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-serviceca\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.303991    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fmf\" (UniqueName: \"kubernetes.io/projected/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-kube-api-access-r6fmf\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304027    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304060    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7ttf\" (UniqueName: \"kubernetes.io/projected/8e96c52d-079e-4b90-b722-b69ad8b12730-kube-api-access-k7ttf\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304087    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304116    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7pwv\" (UniqueName: \"kubernetes.io/projected/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb-kube-api-access-m7pwv\") pod \"kas-connection-checker-685895c7c9-vvw8v\" (UID: \"a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-vvw8v"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304141    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.304235 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304189    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304243    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-netns\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304348    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-kubelet\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304385    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vb9\" (UniqueName: \"kubernetes.io/projected/a56de33d-a1ed-44e6-863f-11ae45cec41f-kube-api-access-95vb9\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304411    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-konnectivity-ca\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304472    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r248d\" (UniqueName: \"kubernetes.io/projected/1a705f33-1517-4f85-8eca-10395ff58b12-kube-api-access-r248d\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304492    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-device-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304516    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d82m\" (UniqueName: \"kubernetes.io/projected/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kube-api-access-9d82m\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.304548 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304535    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-tuned\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304581    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304613    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-tuned\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304635    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304657    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-etc-kubernetes\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304693    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304739    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304745    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2c5\" (UniqueName: \"kubernetes.io/projected/ebc19999-3d20-4929-93a6-cfa85a9074d3-kube-api-access-ln2c5\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304780    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-lib-modules\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.304835 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304810    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304846    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304871    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-sys\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304897    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304901    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.304976    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305005    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305029    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305057    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a705f33-1517-4f85-8eca-10395ff58b12-signing-key\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305087    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-systemd\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305132    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305162    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddgrz\" (UniqueName: \"kubernetes.io/projected/19d1dc0f-27db-4784-916c-dbf422a4d602-kube-api-access-ddgrz\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.305211 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305210    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-service-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305342    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9fdf\" (UniqueName: \"kubernetes.io/projected/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-kube-api-access-w9fdf\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305395    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz48t\" (UniqueName: \"kubernetes.io/projected/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-kube-api-access-dz48t\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305421    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305439    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305458    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-systemd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305487    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305515    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305563    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f92791-c615-4d6e-88e8-1a3f275aa32a-serving-cert\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:59.305644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305639    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-socket-dir-parent\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.305921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305673    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-default-certificate\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.305921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305753    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c6f92791-c615-4d6e-88e8-1a3f275aa32a-serving-cert\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:59.305921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.305900    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-default-certificate\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.307827 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.307809    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 18:36:59.307827 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.307822    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.307956 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.307874    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 18:36:59.307956 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.307885    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 18:36:59.307956 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.307950    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 18:36:59.308105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308010    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 18:36:59.308105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308023    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 18:36:59.308105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308051    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.308105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308050    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.308105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308087    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.308105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308105    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308121    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308161    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308184    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308208    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308228    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308232    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308260    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 18:36:59.308377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308309    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 18:36:59.308621 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308596    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-kp722\""
Oct 30 18:36:59.308909 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308885    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 18:36:59.308909 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308896    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 18:36:59.308909 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308886    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 18:36:59.308909 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.308911    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309809    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309829    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309841    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309859    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309859    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309881    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309893    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309874    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309895    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309913    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309941    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309965    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.309979    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 18:36:59.310000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.310001    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.310700 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.310101    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/daacb5ca-b3fa-4344-add9-c451fc1e5e25-original-pull-secret\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.311039 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.311020    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.311039 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.311036    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9790e2f-d93d-47f4-aa83-6c8a54567513-trusted-ca\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.311138 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.311067    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/334cba69-4d11-4125-9763-d79d806afbc8-nginx-conf\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:59.311179 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.311166    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8e96c52d-079e-4b90-b722-b69ad8b12730-serving-cert\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.312280 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.312267    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-env-overrides\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.312333 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.312267    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:59.313427 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.313410    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.313728 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.313655    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/acca8ca1-d447-4a1f-bd99-8dc48014e033-iptables-alerter-script\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:59.313728 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.313667    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.313900 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.313747    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.313900 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.313747    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-daemon-config\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.314011 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.313952    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.314011 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.313973    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.314122 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.314081    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.314244 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.314225    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.314315 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.314286    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-serviceca\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:59.314315 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.314308    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.314979 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.314965    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:59.315017 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.314989    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.316102 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.316086    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8e96c52d-079e-4b90-b722-b69ad8b12730-service-ca-bundle\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.316161 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.316103    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.316161 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.316106    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.320354 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.320316    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.350812 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.350787    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 18:36:59.350910 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.350892    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-h66ck\""
Oct 30 18:36:59.351918 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.351900    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-script-lib\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.352891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.352877    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sslwz\""
Oct 30 18:36:59.354093 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354080    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j7txc\""
Oct 30 18:36:59.354343 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354317    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-vmn22\""
Oct 30 18:36:59.354490 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354475    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 18:36:59.354541 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354532    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-mrt5r\""
Oct 30 18:36:59.354865 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354846    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.354865 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354845    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 18:36:59.355028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354907    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.355028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354911    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.355028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354929    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.355028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.354954    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 18:36:59.355028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355022    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-thsk7\""
Oct 30 18:36:59.355192 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355050    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 18:36:59.355192 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355126    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-djndp\""
Oct 30 18:36:59.355192 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355154    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hd9r2\""
Oct 30 18:36:59.355192 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355167    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.355192 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355183    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.355442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355206    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 18:36:59.355442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355240    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 18:36:59.355442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355309    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-p85r8\""
Oct 30 18:36:59.355442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355311    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 18:36:59.355442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355379    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4k96h\""
Oct 30 18:36:59.355442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355438    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-88d9s\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355463    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355482    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xm628\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355549    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-kxkbv\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355557    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355581    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355581    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355590    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355596    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355630    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355641    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355629    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 18:36:59.355725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.355686    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6xhn\""
Oct 30 18:36:59.356122 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.356051    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.356122 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.356114    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7jwft\""
Oct 30 18:36:59.356230 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.356121    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 18:36:59.360044 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360015    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 18:36:59.360152 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360090    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5fxhm\""
Oct 30 18:36:59.361009 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360581    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqctd\" (UniqueName: \"kubernetes.io/projected/7d59ccdf-e915-4825-9055-c6b9f4c1c76d-kube-api-access-zqctd\") pod \"migrator-66df5d496b-8lkzz\" (UID: \"7d59ccdf-e915-4825-9055-c6b9f4c1c76d\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz"
Oct 30 18:36:59.361009 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360593    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x95b\" (UniqueName: \"kubernetes.io/projected/acca8ca1-d447-4a1f-bd99-8dc48014e033-kube-api-access-5x95b\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:59.361168 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360611    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.361168 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360654    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-w6mkd\""
Oct 30 18:36:59.361168 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360710    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-bgghq\""
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360780    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dl7q\" (UniqueName: \"kubernetes.io/projected/b9790e2f-d93d-47f4-aa83-6c8a54567513-kube-api-access-8dl7q\") pod \"console-operator-7cd56b8dbc-sfn74\" (UID: \"b9790e2f-d93d-47f4-aa83-6c8a54567513\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360785    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mprt\" (UniqueName: \"kubernetes.io/projected/80b240bd-b789-4575-8a22-f2773e4ef78d-kube-api-access-2mprt\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360821    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8jrs\" (UniqueName: \"kubernetes.io/projected/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-kube-api-access-c8jrs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360832    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-config-volume\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.360950    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a56de33d-a1ed-44e6-863f-11ae45cec41f-cni-binary-copy\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.361072    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46xgz\" (UniqueName: \"kubernetes.io/projected/1e6a72c5-3683-4fc8-8525-e82baed6a4ad-kube-api-access-46xgz\") pod \"kas-connection-checker-685895c7c9-rw5pb\" (UID: \"1e6a72c5-3683-4fc8-8525-e82baed6a4ad\") " pod="kube-system/kas-connection-checker-685895c7c9-rw5pb"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.361070    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a705f33-1517-4f85-8eca-10395ff58b12-signing-cabundle\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.361107    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4njk2\" (UniqueName: \"kubernetes.io/projected/90552f94-1a43-466e-9285-b9e00c4ed560-kube-api-access-4njk2\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.361356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.361266    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhrlf\" (UniqueName: \"kubernetes.io/projected/3ee97fd8-b164-40ca-a9ff-e53e0f461d9e-kube-api-access-mhrlf\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb\" (UID: \"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb"
Oct 30 18:36:59.361655 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.361419    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvfsh\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.361655 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.361478    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjhb\" (UniqueName: \"kubernetes.io/projected/ed959a30-6e00-49b5-aceb-c543a68b9d3e-kube-api-access-ljjhb\") pod \"volume-data-source-validator-58c9f5f46d-rcf9k\" (UID: \"ed959a30-6e00-49b5-aceb-c543a68b9d3e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-rcf9k"
Oct 30 18:36:59.362637 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.362585    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.363008 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.362981    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n59hl\" (UniqueName: \"kubernetes.io/projected/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-kube-api-access-n59hl\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.363088 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.363039    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2s4p\" (UniqueName: \"kubernetes.io/projected/e1321427-04cc-401b-8df9-43b3664b0720-kube-api-access-z2s4p\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:59.363145 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.363130    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r776j\" (UniqueName: \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\") pod \"periodic-gathering-69wwp-9m4cv\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") " pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:36:59.363290 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.363276    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovnkube-config\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.363366 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.363316    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5e89546-8707-4c00-9bf2-826c6fbcd761-ovn-node-metrics-cert\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.363759 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.363743    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/80b240bd-b789-4575-8a22-f2773e4ef78d-cert\") pod \"ingress-canary-dhgxq\" (UID: \"80b240bd-b789-4575-8a22-f2773e4ef78d\") " pod="openshift-ingress-canary/ingress-canary-dhgxq"
Oct 30 18:36:59.364365 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.364345    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-agent-certs\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:59.364679 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.364660    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/511cf18c-8d23-40e7-9e68-f683e3fe9e7d-konnectivity-ca\") pod \"konnectivity-agent-5b6w7\" (UID: \"511cf18c-8d23-40e7-9e68-f683e3fe9e7d\") " pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:36:59.364757 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.364741    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.364921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.364907    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xv4l\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.365449 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.365425    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7pwv\" (UniqueName: \"kubernetes.io/projected/a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb-kube-api-access-m7pwv\") pod \"kas-connection-checker-685895c7c9-vvw8v\" (UID: \"a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb\") " pod="kube-system/kas-connection-checker-685895c7c9-vvw8v"
Oct 30 18:36:59.365586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.365568    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1a705f33-1517-4f85-8eca-10395ff58b12-signing-key\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:59.365656 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.365634    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc7ht\" (UniqueName: \"kubernetes.io/projected/6f0cdf2a-c119-4ffa-8ae0-395f249cf285-kube-api-access-tc7ht\") pod \"network-check-source-6f78d59f99-wf52v\" (UID: \"6f0cdf2a-c119-4ffa-8ae0-395f249cf285\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-wf52v"
Oct 30 18:36:59.365910 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.365892    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh99h\" (UniqueName: \"kubernetes.io/projected/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-kube-api-access-mh99h\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.365910 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.365904    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjnpk\" (UniqueName: \"kubernetes.io/projected/4a159864-3010-4195-84de-1407b16eb5c2-kube-api-access-pjnpk\") pod \"kas-connection-checker-685895c7c9-8cw5s\" (UID: \"4a159864-3010-4195-84de-1407b16eb5c2\") " pod="kube-system/kas-connection-checker-685895c7c9-8cw5s"
Oct 30 18:36:59.366008 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.365944    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fmf\" (UniqueName: \"kubernetes.io/projected/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-kube-api-access-r6fmf\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:59.366643 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.366557    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vb9\" (UniqueName: \"kubernetes.io/projected/a56de33d-a1ed-44e6-863f-11ae45cec41f-kube-api-access-95vb9\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.367002 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.366919    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r248d\" (UniqueName: \"kubernetes.io/projected/1a705f33-1517-4f85-8eca-10395ff58b12-kube-api-access-r248d\") pod \"service-ca-5fffc97bdf-vg6tg\" (UID: \"1a705f33-1517-4f85-8eca-10395ff58b12\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vg6tg"
Oct 30 18:36:59.367090 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.367025    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz48t\" (UniqueName: \"kubernetes.io/projected/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-kube-api-access-dz48t\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.367252 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.367230    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnflj\" (UniqueName: \"kubernetes.io/projected/336c71cc-488f-44cc-8d22-5c8bac5f0bda-kube-api-access-nnflj\") pod \"migrator-66df5d496b-6lp98\" (UID: \"336c71cc-488f-44cc-8d22-5c8bac5f0bda\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98"
Oct 30 18:36:59.367410 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.367391    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2c5\" (UniqueName: \"kubernetes.io/projected/ebc19999-3d20-4929-93a6-cfa85a9074d3-kube-api-access-ln2c5\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:59.367649 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.367625    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7ttf\" (UniqueName: \"kubernetes.io/projected/8e96c52d-079e-4b90-b722-b69ad8b12730-kube-api-access-k7ttf\") pod \"insights-operator-7c4986d449-mx59k\" (UID: \"8e96c52d-079e-4b90-b722-b69ad8b12730\") " pod="openshift-insights/insights-operator-7c4986d449-mx59k"
Oct 30 18:36:59.367756 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.367731    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d82m\" (UniqueName: \"kubernetes.io/projected/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kube-api-access-9d82m\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.367867 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.367849    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58pb\" (UniqueName: \"kubernetes.io/projected/b5e89546-8707-4c00-9bf2-826c6fbcd761-kube-api-access-k58pb\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.368087 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.368058    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddgrz\" (UniqueName: \"kubernetes.io/projected/19d1dc0f-27db-4784-916c-dbf422a4d602-kube-api-access-ddgrz\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.368586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.368561    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9fdf\" (UniqueName: \"kubernetes.io/projected/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-kube-api-access-w9fdf\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.369795 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.369757    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7bmn\" (UniqueName: \"kubernetes.io/projected/d7639481-1e64-49cc-a292-5cfa0cf76416-kube-api-access-l7bmn\") pod \"network-check-target-hktmd\" (UID: \"d7639481-1e64-49cc-a292-5cfa0cf76416\") " pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:36:59.377959 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.377936    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhmtd\" (UniqueName: \"kubernetes.io/projected/c6f92791-c615-4d6e-88e8-1a3f275aa32a-kube-api-access-hhmtd\") pod \"service-ca-operator-544d749c8c-mgvhr\" (UID: \"c6f92791-c615-4d6e-88e8-1a3f275aa32a\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr"
Oct 30 18:36:59.406566 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406541    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-sys\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.406660 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406575    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.406660 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406600    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.406660 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406623    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-systemd\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.406660 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406645    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406659    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-sys\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406670    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-systemd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406690    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406710    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-systemd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406711    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406755    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-socket-dir-parent\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406780    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-node-log\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406804    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406829    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-cnibin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.406862    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.906839784 +0000 UTC m=+1.987425993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : configmap references non-existent config key: service-ca.crt
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406884    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-cnibin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406893    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-kubelet\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.406923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406929    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-socket-dir-parent\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406938    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-netd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406964    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-node-log\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.406965    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-modprobe-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407013    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407037    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-conf\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407060    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90552f94-1a43-466e-9285-b9e00c4ed560-crio-socket\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407083    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-slash\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407105    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-modprobe-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407141    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/90552f94-1a43-466e-9285-b9e00c4ed560-crio-socket\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407175    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/90552f94-1a43-466e-9285-b9e00c4ed560-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407211    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-slash\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407220    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407235    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-systemd\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407248    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-kubelet\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407276    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-netd\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407283    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407109    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-d\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407382    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysctl-conf\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407385    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407443    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.407467    5025 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407515    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-conf-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.407528    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.907512406 +0000 UTC m=+1.988098611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407468    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-multus-conf-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407536    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-sys-fs\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.407546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407560    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-netns\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407593    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysconfig\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407618    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-host\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407642    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-systemd-units\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407645    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-netns\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407664    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-k8s-cni-cncf-io\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407669    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-sysconfig\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407697    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-host\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407731    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-systemd-units\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407754    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-var-lib-kubelet\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407790    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-k8s-cni-cncf-io\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407805    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-log-socket\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407837    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407841    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-var-lib-kubelet\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.407948    5025 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.407962    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-log-socket\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.407994    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.907980922 +0000 UTC m=+1.988567126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408024    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408049    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-var-lib-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408102    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-registration-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408111    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-etc-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408136    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408144    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-var-lib-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408169    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-run\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408192    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408187    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-etc-openvswitch\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408261    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-kubernetes\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.408288    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.408299    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408344    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-run\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.408368    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.90835616 +0000 UTC m=+1.988942384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408216    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-etc-kubernetes\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408397    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-multus\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408436    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408440    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-multus\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408403    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-socket-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408461    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cnibin\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408496    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-host\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.408522    5025 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408552    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-bin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408520    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-cni-bin\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.408566    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.90855212 +0000 UTC m=+1.989138331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408606    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-cnibin\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408609    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9c90acf4-d9a8-4478-a97d-cc5bfe500fa0-host\") pod \"node-ca-zfktp\" (UID: \"9c90acf4-d9a8-4478-a97d-cc5bfe500fa0\") " pod="openshift-image-registry/node-ca-zfktp"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408647    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-multus-certs\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408670    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-bin\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.408686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408695    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408724    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-multus-certs\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408731    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408760    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408768    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-cni-bin\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408784    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-dbus\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408829    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-hostroot\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408835    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-etc-selinux\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408852    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-system-cni-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408891    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-kubelet-config\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.408903    5025 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408913    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.408944    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls podName:c4944e38-f0de-40f8-88f3-26fc7b9c2c63 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.908931614 +0000 UTC m=+1.989517820 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-djc4m" (UID: "c4944e38-f0de-40f8-88f3-26fc7b9c2c63") : secret "samples-operator-tls" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408962    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408987    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.408993    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acca8ca1-d447-4a1f-bd99-8dc48014e033-host-slash\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409017    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebc19999-3d20-4929-93a6-cfa85a9074d3-hosts-file\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409043    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-system-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409063    5025 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409070    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-ovn\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409110    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.90908586 +0000 UTC m=+1.989672077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409114    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-run-ovn\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409129    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409159    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5e89546-8707-4c00-9bf2-826c6fbcd761-host-run-ovn-kubernetes\") pod \"ovnkube-node-5lk89\" (UID: \"b5e89546-8707-4c00-9bf2-826c6fbcd761\") " pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409155    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409201    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/acca8ca1-d447-4a1f-bd99-8dc48014e033-host-slash\") pod \"iptables-alerter-fcsph\" (UID: \"acca8ca1-d447-4a1f-bd99-8dc48014e033\") " pod="openshift-network-operator/iptables-alerter-fcsph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409203    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409235    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-netns\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409261    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-kubelet\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409275    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409285    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409288    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-device-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409314    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.90930414 +0000 UTC m=+1.989890361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409349    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-etc-kubernetes\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409365    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-device-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409376    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-lib-modules\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409413    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ebc19999-3d20-4929-93a6-cfa85a9074d3-hosts-file\") pod \"node-resolver-s6975\" (UID: \"ebc19999-3d20-4929-93a6-cfa85a9074d3\") " pod="openshift-dns/node-resolver-s6975"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409415    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409466    5025 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409500    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-system-cni-dir\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409510    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs podName:41314e0f-aadd-4cdf-aac1-93fb90cb9a5f nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.909498495 +0000 UTC m=+1.990084716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs") pod "router-default-b457d476c-cmkxg" (UID: "41314e0f-aadd-4cdf-aac1-93fb90cb9a5f") : secret "router-metrics-certs-default" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409538    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-run-netns\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409572    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-dbus\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409663    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d51c59f-38ab-4311-bf40-cf79edf28d4a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7nd67\" (UID: \"9d51c59f-38ab-4311-bf40-cf79edf28d4a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7nd67"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409695    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/19d1dc0f-27db-4784-916c-dbf422a4d602-lib-modules\") pod \"tuned-m8v44\" (UID: \"19d1dc0f-27db-4784-916c-dbf422a4d602\") " pod="openshift-cluster-node-tuning-operator/tuned-m8v44"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409575    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-host-var-lib-kubelet\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409739    5025 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.409771    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:36:59.909759267 +0000 UTC m=+1.990345494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : secret "metrics-daemon-secret" not found
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409803    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-hostroot\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409854    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77bb5229-a381-4bf0-82ee-e4eb959cf7eb-system-cni-dir\") pod \"multus-additional-cni-plugins-m4dhb\" (UID: \"77bb5229-a381-4bf0-82ee-e4eb959cf7eb\") " pod="openshift-multus/multus-additional-cni-plugins-m4dhb"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409896    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/daacb5ca-b3fa-4344-add9-c451fc1e5e25-kubelet-config\") pod \"global-pull-secret-syncer-k8tk8\" (UID: \"daacb5ca-b3fa-4344-add9-c451fc1e5e25\") " pod="kube-system/global-pull-secret-syncer-k8tk8"
Oct 30 18:36:59.410465 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.409933    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a56de33d-a1ed-44e6-863f-11ae45cec41f-etc-kubernetes\") pod \"multus-2kfph\" (UID: \"a56de33d-a1ed-44e6-863f-11ae45cec41f\") " pod="openshift-multus/multus-2kfph"
Oct 30 18:36:59.504250 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.504221    5025 scope.go:122] "RemoveContainer" containerID="826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870"
Oct 30 18:36:59.520069 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.520052    5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:36:59.774506 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.774432    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:36:59.782119 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.782098    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zfktp_9c90acf4-d9a8-4478-a97d-cc5bfe500fa0/node-ca/0.log"
Oct 30 18:36:59.913591 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.913559    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:36:59.913737 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.913597    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:36:59.913737 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.913622    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.913737 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.913641    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.913737 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.913683    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:36:59.913737 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913704    5025 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913767    5025 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.913708    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913771    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:00.913754091 +0000 UTC m=+2.994340296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : secret "metrics-daemon-secret" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913772    5025 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913711    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913866    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:00.913853047 +0000 UTC m=+2.994439238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913872    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913890    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:00.913880941 +0000 UTC m=+2.994467129 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.913927    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:00.913911929 +0000 UTC m=+2.994498222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.913965    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.914003    5025 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.914031    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:36:59.914053 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.914048    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.914065    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.914068    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.914104    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:00.914091687 +0000 UTC m=+2.994677898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.914125    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.914152    5025 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.914212    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:37:00.914197196 +0000 UTC m=+2.994783393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.914236    5025 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:36:59.914279    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:00.914265623 +0000 UTC m=+2.994851820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:36:59.914618 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.914420    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-service-ca-bundle\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.916389 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.916363    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c4944e38-f0de-40f8-88f3-26fc7b9c2c63-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-djc4m\" (UID: \"c4944e38-f0de-40f8-88f3-26fc7b9c2c63\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:36:59.916555 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.916542    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41314e0f-aadd-4cdf-aac1-93fb90cb9a5f-metrics-certs\") pod \"router-default-b457d476c-cmkxg\" (UID: \"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f\") " pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:36:59.990113 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:36:59.990085    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:37:00.047720 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.047640    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:00.105093 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.105070    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:37:00.107184 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.107166    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"
Oct 30 18:37:00.136165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.133779    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:37:00.136165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.134248    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/1.log"
Oct 30 18:37:00.136165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.134276    5025 generic.go:356] "Generic (PLEG): container finished" podID="b9790e2f-d93d-47f4-aa83-6c8a54567513" containerID="be40be794f7c6b97533e49ec943e2f7147ca3ed8ab442c7640dd6afc264e79da" exitCode=255
Oct 30 18:37:00.136165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.134841    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" event={"ID":"b9790e2f-d93d-47f4-aa83-6c8a54567513","Type":"ContainerDied","Data":"be40be794f7c6b97533e49ec943e2f7147ca3ed8ab442c7640dd6afc264e79da"}
Oct 30 18:37:00.136165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.134899    5025 scope.go:122] "RemoveContainer" containerID="826bac9586156a3abecc751b99a6c6bdeb52d157a69557b2bc1565388e664870"
Oct 30 18:37:00.136165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.136074    5025 scope.go:122] "RemoveContainer" containerID="be40be794f7c6b97533e49ec943e2f7147ca3ed8ab442c7640dd6afc264e79da"
Oct 30 18:37:00.136481 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.136217    5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sfn74_openshift-console-operator(b9790e2f-d93d-47f4-aa83-6c8a54567513)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" podUID="b9790e2f-d93d-47f4-aa83-6c8a54567513"
Oct 30 18:37:00.195562 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.195531    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-dhgxq_80b240bd-b789-4575-8a22-f2773e4ef78d/serve-healthcheck-canary/0.log"
Oct 30 18:37:00.554399 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.554374    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m"]
Oct 30 18:37:00.557408 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.557378    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b457d476c-cmkxg"]
Oct 30 18:37:00.591805 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:00.591773    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41314e0f_aadd_4cdf_aac1_93fb90cb9a5f.slice/crio-8316056037b7f2f269301da72c7bf338aa1fa6f197f5e2932478966b3fc72934 WatchSource:0}: Error finding container 8316056037b7f2f269301da72c7bf338aa1fa6f197f5e2932478966b3fc72934: Status 404 returned error can't find the container with id 8316056037b7f2f269301da72c7bf338aa1fa6f197f5e2932478966b3fc72934
Oct 30 18:37:00.782681 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.782650    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb_3ee97fd8-b164-40ca-a9ff-e53e0f461d9e/kube-storage-version-migrator-operator/0.log"
Oct 30 18:37:00.862906 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.862845    5025 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:37:00.922569 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.922542    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:37:00.922704 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.922614    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:37:00.922704 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.922640    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:37:00.922704 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922664    5025 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922717    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.922703992 +0000 UTC m=+5.003290210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.922713    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922758    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922769    5025 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922801    5025 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922811    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls podName:90552f94-1a43-466e-9285-b9e00c4ed560 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.922800641 +0000 UTC m=+5.003386863 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls") pod "insights-runtime-extractor-d52ch" (UID: "90552f94-1a43-466e-9285-b9e00c4ed560") : secret "insights-runtime-extractor-tls" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922816    5025 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.922759    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922828    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.922821108 +0000 UTC m=+5.003407333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:37:00.922837 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922773    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922857    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs podName:e1321427-04cc-401b-8df9-43b3664b0720 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.922845835 +0000 UTC m=+5.003432026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs") pod "network-metrics-daemon-fv5vj" (UID: "e1321427-04cc-401b-8df9-43b3664b0720") : secret "metrics-daemon-secret" not found
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.922883    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922897    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.922886865 +0000 UTC m=+5.003473054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:00.922930    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922936    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922945    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922967    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.922960072 +0000 UTC m=+5.003546256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.922998    5025 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 18:37:00.923117 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:00.923026    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls podName:e487a2e3-2c52-4dea-80a6-fdb09e9229ae nodeName:}" failed. No retries permitted until 2026-10-30 18:37:02.92301932 +0000 UTC m=+5.003605505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-4gt4s" (UID: "e487a2e3-2c52-4dea-80a6-fdb09e9229ae") : secret "cluster-monitoring-operator-tls" not found
Oct 30 18:37:01.140532 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.140507    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6lp98" event={"ID":"336c71cc-488f-44cc-8d22-5c8bac5f0bda","Type":"ContainerStarted","Data":"5658eae84543c88a8d5fb665bb6d8a18d86af2f8d47c8904a441ad71ebc041aa"}
Oct 30 18:37:01.142136 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.142113    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:37:01.142438 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.142412    5025 scope.go:122] "RemoveContainer" containerID="be40be794f7c6b97533e49ec943e2f7147ca3ed8ab442c7640dd6afc264e79da"
Oct 30 18:37:01.142627 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:01.142591    5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sfn74_openshift-console-operator(b9790e2f-d93d-47f4-aa83-6c8a54567513)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" podUID="b9790e2f-d93d-47f4-aa83-6c8a54567513"
Oct 30 18:37:01.145547 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.145523    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b457d476c-cmkxg" event={"ID":"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f","Type":"ContainerStarted","Data":"3bb36526ba4d6e428def5bd504b4d2f84fd2c4b723b4567ff86c1f554800ac72"}
Oct 30 18:37:01.145632 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.145565    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b457d476c-cmkxg" event={"ID":"41314e0f-aadd-4cdf-aac1-93fb90cb9a5f","Type":"ContainerStarted","Data":"8316056037b7f2f269301da72c7bf338aa1fa6f197f5e2932478966b3fc72934"}
Oct 30 18:37:01.146779 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.146757    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m" event={"ID":"c4944e38-f0de-40f8-88f3-26fc7b9c2c63","Type":"ContainerStarted","Data":"e46c16c17248b37722bd495112785b16263ab6814bb97806d5412f7bf9ed2526"}
Oct 30 18:37:01.199570 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.199541    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:01.418164 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.418070    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:01.661378 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.661343    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:01.876069 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:01.876031    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:02.091777 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.091729    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:02.105531 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.105505    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:37:02.108334 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.108301    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:37:02.153178 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.153090    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-8lkzz" event={"ID":"7d59ccdf-e915-4825-9055-c6b9f4c1c76d","Type":"ContainerStarted","Data":"97a492030882e3373d66e2ea4a1793da0efe3fe88cb74fdb48dabd24baad7ab6"}
Oct 30 18:37:02.471972 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.471912    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:37:02.942486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.942446    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:37:02.942694 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.942517    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:37:02.942694 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.942538    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:37:02.942694 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.942593    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:37:02.942694 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942618    5025 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 18:37:02.942694 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942674    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:37:02.942694 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942681    5025 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:37:02.942694 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942693    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-b895475ff-2t4mb: secret "image-registry-tls" not found
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.942627    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942701    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert podName:334cba69-4d11-4125-9763-d79d806afbc8 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:06.942680783 +0000 UTC m=+9.023266983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-s4bss" (UID: "334cba69-4d11-4125-9763-d79d806afbc8") : secret "networking-console-plugin-cert" not found
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942734    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:06.942718917 +0000 UTC m=+9.023305119 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942753    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls podName:8d964528-47c1-4c62-8185-a27cc4cb8447 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:06.9427451 +0000 UTC m=+9.023331286 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls") pod "image-registry-b895475ff-2t4mb" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447") : secret "image-registry-tls" not found
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.942784    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.942816    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942908    5025 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942923    5025 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d9cdb74b-n2d7w: secret "image-registry-tls" not found
Oct 30 18:37:02.942999 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:02.942971    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls podName:fbba07e7-11e1-4277-96ff-0a56c96c9ac1 nodeName:}" failed. No retries permitted until 2026-10-30 18:37:06.94295719 +0000 UTC m=+9.023543398 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls") pod "image-registry-8d9cdb74b-n2d7w" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1") : secret "image-registry-tls" not found
Oct 30 18:37:02.944853 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.944832    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1321427-04cc-401b-8df9-43b3664b0720-metrics-certs\") pod \"network-metrics-daemon-fv5vj\" (UID: \"e1321427-04cc-401b-8df9-43b3664b0720\") " pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:37:02.945007 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.944987    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/90552f94-1a43-466e-9285-b9e00c4ed560-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d52ch\" (UID: \"90552f94-1a43-466e-9285-b9e00c4ed560\") " pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:37:02.945054 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:02.945037    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e487a2e3-2c52-4dea-80a6-fdb09e9229ae-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-4gt4s\" (UID: \"e487a2e3-2c52-4dea-80a6-fdb09e9229ae\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:37:03.085624 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.085576    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:03.107740 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.107715    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"
Oct 30 18:37:03.112297 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.112277    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fv5vj"
Oct 30 18:37:03.120349 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.120316    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d52ch"
Oct 30 18:37:03.157476 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.157443    5025 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:37:03.345557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.345525    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:03.518430 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.518347    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d52ch"]
Oct 30 18:37:03.551094 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:03.551064    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90552f94_1a43_466e_9285_b9e00c4ed560.slice/crio-f609003036d9011d1ce2b92426144db3ba4dc9fa21aaf602de9032fe2ce84603 WatchSource:0}: Error finding container f609003036d9011d1ce2b92426144db3ba4dc9fa21aaf602de9032fe2ce84603: Status 404 returned error can't find the container with id f609003036d9011d1ce2b92426144db3ba4dc9fa21aaf602de9032fe2ce84603
Oct 30 18:37:03.558475 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.558451    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fv5vj"]
Oct 30 18:37:03.561706 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.561685    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s"]
Oct 30 18:37:03.565371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.565347    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:03.571922 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:03.571903    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1321427_04cc_401b_8df9_43b3664b0720.slice/crio-9d6d33906d1e7e32dc06093400974f8b98d2f4264ba93d8f958a8dc4185e6b22 WatchSource:0}: Error finding container 9d6d33906d1e7e32dc06093400974f8b98d2f4264ba93d8f958a8dc4185e6b22: Status 404 returned error can't find the container with id 9d6d33906d1e7e32dc06093400974f8b98d2f4264ba93d8f958a8dc4185e6b22
Oct 30 18:37:03.601034 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:03.601008    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode487a2e3_2c52_4dea_80a6_fdb09e9229ae.slice/crio-7a62750a9ef33c7f3b916bdaceb34bbb22154af755b7dad898bcedbc5052ad7f WatchSource:0}: Error finding container 7a62750a9ef33c7f3b916bdaceb34bbb22154af755b7dad898bcedbc5052ad7f: Status 404 returned error can't find the container with id 7a62750a9ef33c7f3b916bdaceb34bbb22154af755b7dad898bcedbc5052ad7f
Oct 30 18:37:03.783264 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:03.783181    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:04.015262 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:04.015228    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:04.162056 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:04.162011    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s" event={"ID":"e487a2e3-2c52-4dea-80a6-fdb09e9229ae","Type":"ContainerStarted","Data":"7a62750a9ef33c7f3b916bdaceb34bbb22154af755b7dad898bcedbc5052ad7f"}
Oct 30 18:37:04.163685 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:04.163650    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d52ch" event={"ID":"90552f94-1a43-466e-9285-b9e00c4ed560","Type":"ContainerStarted","Data":"a0d5f10bc2ea36242e079014565bf88dfd3a9f06f6c5406b1b88c2eba4941ee0"}
Oct 30 18:37:04.163814 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:04.163709    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d52ch" event={"ID":"90552f94-1a43-466e-9285-b9e00c4ed560","Type":"ContainerStarted","Data":"f609003036d9011d1ce2b92426144db3ba4dc9fa21aaf602de9032fe2ce84603"}
Oct 30 18:37:04.164851 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:04.164825    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fv5vj" event={"ID":"e1321427-04cc-401b-8df9-43b3664b0720","Type":"ContainerStarted","Data":"9d6d33906d1e7e32dc06093400974f8b98d2f4264ba93d8f958a8dc4185e6b22"}
Oct 30 18:37:04.166422 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:04.166399    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m" event={"ID":"c4944e38-f0de-40f8-88f3-26fc7b9c2c63","Type":"ContainerStarted","Data":"6072c1dfd3ded8646ecf1eb17e5aceb3fe256fe79109a7d61590c402abb5ef71"}
Oct 30 18:37:04.166532 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:04.166439    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-djc4m" event={"ID":"c4944e38-f0de-40f8-88f3-26fc7b9c2c63","Type":"ContainerStarted","Data":"bf94511db3eb8d2bfa6b93daa2a0757132eb9aceebd1aec4abf5313349211077"}
Oct 30 18:37:05.172037 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:05.172003    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d52ch" event={"ID":"90552f94-1a43-466e-9285-b9e00c4ed560","Type":"ContainerStarted","Data":"fa41819465f9523f43e24302e10a8bf36c3b01513d11fa4d03457d6ea1f89a35"}
Oct 30 18:37:05.182463 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:05.182438    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:37:05.182584 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:05.182551    5025 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:37:05.183924 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:05.183904    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b457d476c-cmkxg"
Oct 30 18:37:05.188768 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:05.188740    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:06.177998 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.177968    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fv5vj" event={"ID":"e1321427-04cc-401b-8df9-43b3664b0720","Type":"ContainerStarted","Data":"0a7849f0eac33e3f8fe603eac458e3726382b91b9a39aaa07c344ea5041d8e3c"}
Oct 30 18:37:06.178410 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.178032    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fv5vj" event={"ID":"e1321427-04cc-401b-8df9-43b3664b0720","Type":"ContainerStarted","Data":"26b66c150927cd1cf8a59bc81e074dc18054ccdfeda7b79a9501138d43583fb9"}
Oct 30 18:37:06.981827 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.981738    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:37:06.981827 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.981808    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:37:06.982054 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.981884    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:06.982054 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.981910    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:06.982054 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:06.982006    5025 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 18:37:06.982203 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:06.982072    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls podName:accf98f1-25fe-4fb4-b63a-05ad93a65d1f nodeName:}" failed. No retries permitted until 2026-10-30 18:37:14.982057458 +0000 UTC m=+17.062643643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls") pod "dns-default-gbsgk" (UID: "accf98f1-25fe-4fb4-b63a-05ad93a65d1f") : secret "dns-default-metrics-tls" not found
Oct 30 18:37:06.984388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.984360    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"image-registry-b895475ff-2t4mb\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") " pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:37:06.984526 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.984425    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"image-registry-8d9cdb74b-n2d7w\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") " pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:06.984631 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:06.984609    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/334cba69-4d11-4125-9763-d79d806afbc8-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-s4bss\" (UID: \"334cba69-4d11-4125-9763-d79d806afbc8\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:37:07.008486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.008465    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:07.008604 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.008512    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss"
Oct 30 18:37:07.015541 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.015526    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:37:07.111142 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.109857    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:37:07.111142 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.110028    5025 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:37:07.183089 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.183055    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-4gt4s" event={"ID":"e487a2e3-2c52-4dea-80a6-fdb09e9229ae","Type":"ContainerStarted","Data":"31eaa7be8337bca94437afc2317b296c6517681f0e7df8d341522a7610e6f282"}
Oct 30 18:37:07.185245 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.185212    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d52ch" event={"ID":"90552f94-1a43-466e-9285-b9e00c4ed560","Type":"ContainerStarted","Data":"6554358827dc8a9652948e890951d3f1416b5411927e18d3c9ecf09c0694c213"}
Oct 30 18:37:07.200904 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.200872    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:07.410766 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.410732    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:07.444851 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.444825    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-s4bss"]
Oct 30 18:37:07.450012 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.449988    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-b895475ff-2t4mb"]
Oct 30 18:37:07.450732 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.450714    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d9cdb74b-n2d7w"]
Oct 30 18:37:07.452545 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:07.452522    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334cba69_4d11_4125_9763_d79d806afbc8.slice/crio-ad71716cca67af4f955745f8fccca58fac78f2c5bc7b53d8fdd4a1ebb71ffb6e WatchSource:0}: Error finding container ad71716cca67af4f955745f8fccca58fac78f2c5bc7b53d8fdd4a1ebb71ffb6e: Status 404 returned error can't find the container with id ad71716cca67af4f955745f8fccca58fac78f2c5bc7b53d8fdd4a1ebb71ffb6e
Oct 30 18:37:07.455716 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:07.455694    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbba07e7_11e1_4277_96ff_0a56c96c9ac1.slice/crio-d8aa3d9905f1a0943cbaac54afbab949087bb89a6604bcd0738d236f4c005c6c WatchSource:0}: Error finding container d8aa3d9905f1a0943cbaac54afbab949087bb89a6604bcd0738d236f4c005c6c: Status 404 returned error can't find the container with id d8aa3d9905f1a0943cbaac54afbab949087bb89a6604bcd0738d236f4c005c6c
Oct 30 18:37:07.456515 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:07.456494    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d964528_47c1_4c62_8185_a27cc4cb8447.slice/crio-47a3d724b291dd3512138489acd9f1cd969e323b7eca486ba031685507a0a2bc WatchSource:0}: Error finding container 47a3d724b291dd3512138489acd9f1cd969e323b7eca486ba031685507a0a2bc: Status 404 returned error can't find the container with id 47a3d724b291dd3512138489acd9f1cd969e323b7eca486ba031685507a0a2bc
Oct 30 18:37:07.641433 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.641391    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:07.816352 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.816267    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5b6w7"
Oct 30 18:37:07.862831 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:07.862803    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:08.027585 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.027557    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:37:08.027980 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.027963    5025 scope.go:122] "RemoveContainer" containerID="be40be794f7c6b97533e49ec943e2f7147ca3ed8ab442c7640dd6afc264e79da"
Oct 30 18:37:08.028195 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:08.028173    5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-sfn74_openshift-console-operator(b9790e2f-d93d-47f4-aa83-6c8a54567513)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" podUID="b9790e2f-d93d-47f4-aa83-6c8a54567513"
Oct 30 18:37:08.056094 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.056067    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:08.190665 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.190632    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss" event={"ID":"334cba69-4d11-4125-9763-d79d806afbc8","Type":"ContainerStarted","Data":"ad71716cca67af4f955745f8fccca58fac78f2c5bc7b53d8fdd4a1ebb71ffb6e"}
Oct 30 18:37:08.192119 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.192093    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" event={"ID":"8d964528-47c1-4c62-8185-a27cc4cb8447","Type":"ContainerStarted","Data":"989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e"}
Oct 30 18:37:08.192210 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.192131    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" event={"ID":"8d964528-47c1-4c62-8185-a27cc4cb8447","Type":"ContainerStarted","Data":"47a3d724b291dd3512138489acd9f1cd969e323b7eca486ba031685507a0a2bc"}
Oct 30 18:37:08.193662 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.193637    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w" event={"ID":"fbba07e7-11e1-4277-96ff-0a56c96c9ac1","Type":"ContainerStarted","Data":"df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6"}
Oct 30 18:37:08.193774 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.193677    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w" event={"ID":"fbba07e7-11e1-4277-96ff-0a56c96c9ac1","Type":"ContainerStarted","Data":"d8aa3d9905f1a0943cbaac54afbab949087bb89a6604bcd0738d236f4c005c6c"}
Oct 30 18:37:08.416688 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.416652    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:08.979451 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:08.979417    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:09.199921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:09.199893    5025 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:37:09.199921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:09.199897    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-s4bss" event={"ID":"334cba69-4d11-4125-9763-d79d806afbc8","Type":"ContainerStarted","Data":"6779c4a458a01d113f046e25e50f6cffea88cde29defab6b479ddb2f35311410"}
Oct 30 18:37:09.200363 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:09.200260    5025 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 18:37:09.414644 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:09.414608    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:09.672233 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:09.672156    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:09.873854 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:09.873824    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:10.050472 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.050406    5025 scope.go:122] "RemoveContainer" containerID="be40be794f7c6b97533e49ec943e2f7147ca3ed8ab442c7640dd6afc264e79da"
Oct 30 18:37:10.181129 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.181093    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:10.205237 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.205216    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:37:10.205605 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.205342    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" event={"ID":"b9790e2f-d93d-47f4-aa83-6c8a54567513","Type":"ContainerStarted","Data":"615f8f274635a5a239dbba9ccbab7c9d395b82419af4492c3f3e493b5f99b475"}
Oct 30 18:37:10.205661 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.205609    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:37:10.206645 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.206624    5025 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-sfn74 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body=
Oct 30 18:37:10.206747 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.206669    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74" podUID="b9790e2f-d93d-47f4-aa83-6c8a54567513" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused"
Oct 30 18:37:10.375872 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:10.375841    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.212521 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:11.212494    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-sfn74"
Oct 30 18:37:11.562115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:11.562027    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.780656 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:11.780623    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:11.924846 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:11.924818    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:37:12.084911 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:12.084877    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:12.373397 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:12.373360    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:15.055019 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:15.054984    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:15.057228 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:15.057202    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/accf98f1-25fe-4fb4-b63a-05ad93a65d1f-metrics-tls\") pod \"dns-default-gbsgk\" (UID: \"accf98f1-25fe-4fb4-b63a-05ad93a65d1f\") " pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:15.116491 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:15.116465    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:15.484829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:15.484801    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gbsgk"]
Oct 30 18:37:15.490906 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:15.490882    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccf98f1_25fe_4fb4_b63a_05ad93a65d1f.slice/crio-0702e3aa2c9515c0cafc083a2358989978c73c6b9454289b240ef402a6ee69bc WatchSource:0}: Error finding container 0702e3aa2c9515c0cafc083a2358989978c73c6b9454289b240ef402a6ee69bc: Status 404 returned error can't find the container with id 0702e3aa2c9515c0cafc083a2358989978c73c6b9454289b240ef402a6ee69bc
Oct 30 18:37:15.690470 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:15.690434    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:16.213390 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:16.213351    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:16.229340 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:16.229294    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbsgk" event={"ID":"accf98f1-25fe-4fb4-b63a-05ad93a65d1f","Type":"ContainerStarted","Data":"0702e3aa2c9515c0cafc083a2358989978c73c6b9454289b240ef402a6ee69bc"}
Oct 30 18:37:16.481642 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:16.481527    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:17.928187 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:17.928154    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"]
Oct 30 18:37:17.931573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:17.931518    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"
Oct 30 18:37:17.982588 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:17.982560    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72dd3f31-31ed-41a4-a097-8d01f4063960-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fhbbl\" (UID: \"72dd3f31-31ed-41a4-a097-8d01f4063960\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"
Oct 30 18:37:17.989521 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:17.989502    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-4ljfg\""
Oct 30 18:37:18.001315 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.001292    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 18:37:18.037840 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.037813    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8d9cdb74b-n2d7w"]
Oct 30 18:37:18.038666 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.038649    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-6td24"]
Oct 30 18:37:18.042047 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.042030    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.083076 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.083045    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4vjg\" (UniqueName: \"kubernetes.io/projected/31d8d46c-f388-43a2-9386-26166b60f904-kube-api-access-d4vjg\") pod \"downloads-6dd789c8b7-6td24\" (UID: \"31d8d46c-f388-43a2-9386-26166b60f904\") " pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.083220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.083089    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72dd3f31-31ed-41a4-a097-8d01f4063960-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fhbbl\" (UID: \"72dd3f31-31ed-41a4-a097-8d01f4063960\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"
Oct 30 18:37:18.083220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.083119    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d8d46c-f388-43a2-9386-26166b60f904-tmp\") pod \"downloads-6dd789c8b7-6td24\" (UID: \"31d8d46c-f388-43a2-9386-26166b60f904\") " pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.085435 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.085414    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/72dd3f31-31ed-41a4-a097-8d01f4063960-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-fhbbl\" (UID: \"72dd3f31-31ed-41a4-a097-8d01f4063960\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"
Oct 30 18:37:18.098296 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.098258    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 18:37:18.098411 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.098367    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-l298j\""
Oct 30 18:37:18.098411 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.098383    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 18:37:18.134634 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.134610    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-c4cbdd774-cfpcv"]
Oct 30 18:37:18.138115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.138096    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184660 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184589    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d8d46c-f388-43a2-9386-26166b60f904-tmp\") pod \"downloads-6dd789c8b7-6td24\" (UID: \"31d8d46c-f388-43a2-9386-26166b60f904\") " pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.184660 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184627    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f85b88d9-6748-43f2-a172-8377a7f8e700-image-registry-private-configuration\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184660 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184655    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-registry-tls\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184853 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184671    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f85b88d9-6748-43f2-a172-8377a7f8e700-registry-certificates\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184853 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184762    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-bound-sa-token\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184853 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184804    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f85b88d9-6748-43f2-a172-8377a7f8e700-installation-pull-secrets\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184853 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184833    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4vjg\" (UniqueName: \"kubernetes.io/projected/31d8d46c-f388-43a2-9386-26166b60f904-kube-api-access-d4vjg\") pod \"downloads-6dd789c8b7-6td24\" (UID: \"31d8d46c-f388-43a2-9386-26166b60f904\") " pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.184972 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184885    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f85b88d9-6748-43f2-a172-8377a7f8e700-trusted-ca\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184972 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184914    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f85b88d9-6748-43f2-a172-8377a7f8e700-ca-trust-extracted\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.184972 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184939    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghq6l\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-kube-api-access-ghq6l\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.185062 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.184979    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31d8d46c-f388-43a2-9386-26166b60f904-tmp\") pod \"downloads-6dd789c8b7-6td24\" (UID: \"31d8d46c-f388-43a2-9386-26166b60f904\") " pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.239773 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.239737    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbsgk" event={"ID":"accf98f1-25fe-4fb4-b63a-05ad93a65d1f","Type":"ContainerStarted","Data":"9e3b6e40d8868edfb802c8cb2be4249d6a579303530f855ab60bd24779b91dc3"}
Oct 30 18:37:18.239931 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.239798    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gbsgk" event={"ID":"accf98f1-25fe-4fb4-b63a-05ad93a65d1f","Type":"ContainerStarted","Data":"baeee75a76bc9fa657a7ecd93f6cff1a14ac722a184c0c71f8edcb0327339bc9"}
Oct 30 18:37:18.239931 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.239906    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:18.242311 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.242273    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"
Oct 30 18:37:18.242730 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.242694    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:18.245371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.245351    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4vjg\" (UniqueName: \"kubernetes.io/projected/31d8d46c-f388-43a2-9386-26166b60f904-kube-api-access-d4vjg\") pod \"downloads-6dd789c8b7-6td24\" (UID: \"31d8d46c-f388-43a2-9386-26166b60f904\") " pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.285615 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.285583    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghq6l\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-kube-api-access-ghq6l\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.285786 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.285629    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f85b88d9-6748-43f2-a172-8377a7f8e700-image-registry-private-configuration\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.285949 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.285923    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-registry-tls\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.286019 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.285968    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f85b88d9-6748-43f2-a172-8377a7f8e700-registry-certificates\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.286082 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.286018    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-bound-sa-token\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.286082 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.286066    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f85b88d9-6748-43f2-a172-8377a7f8e700-installation-pull-secrets\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.286188 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.286133    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f85b88d9-6748-43f2-a172-8377a7f8e700-trusted-ca\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.286188 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.286165    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f85b88d9-6748-43f2-a172-8377a7f8e700-ca-trust-extracted\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.286634 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.286612    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f85b88d9-6748-43f2-a172-8377a7f8e700-ca-trust-extracted\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.287236 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.287213    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f85b88d9-6748-43f2-a172-8377a7f8e700-registry-certificates\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.287362 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.287270    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f85b88d9-6748-43f2-a172-8377a7f8e700-trusted-ca\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.288368 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.288345    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f85b88d9-6748-43f2-a172-8377a7f8e700-image-registry-private-configuration\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.289002 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.288975    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f85b88d9-6748-43f2-a172-8377a7f8e700-installation-pull-secrets\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.289981 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.289958    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-registry-tls\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.353762 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.353742    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:18.359169 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.359142    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghq6l\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-kube-api-access-ghq6l\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.363793 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.363765    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f85b88d9-6748-43f2-a172-8377a7f8e700-bound-sa-token\") pod \"image-registry-c4cbdd774-cfpcv\" (UID: \"f85b88d9-6748-43f2-a172-8377a7f8e700\") " pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.452121 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.452054    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:18.492061 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.492027    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c4cbdd774-cfpcv"]
Oct 30 18:37:18.513112 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.513075    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"]
Oct 30 18:37:18.565204 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.565177    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6td24"]
Oct 30 18:37:18.636037 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.635996    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:18.919481 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:18.919450    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:19.019917 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.019852    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"]
Oct 30 18:37:19.026992 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:19.026968    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dd3f31_31ed_41a4_a097_8d01f4063960.slice/crio-003c1ef99612e08e6db1eb054359efcef4d37ee88c008f23add706f4adcdfc52 WatchSource:0}: Error finding container 003c1ef99612e08e6db1eb054359efcef4d37ee88c008f23add706f4adcdfc52: Status 404 returned error can't find the container with id 003c1ef99612e08e6db1eb054359efcef4d37ee88c008f23add706f4adcdfc52
Oct 30 18:37:19.071582 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.071548    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-c4cbdd774-cfpcv"]
Oct 30 18:37:19.077504 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:19.077473    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85b88d9_6748_43f2_a172_8377a7f8e700.slice/crio-efa790c6f81707dceaebc4c0d03e73ef0d1b9dffffcb6048ec7c30b239949f0b WatchSource:0}: Error finding container efa790c6f81707dceaebc4c0d03e73ef0d1b9dffffcb6048ec7c30b239949f0b: Status 404 returned error can't find the container with id efa790c6f81707dceaebc4c0d03e73ef0d1b9dffffcb6048ec7c30b239949f0b
Oct 30 18:37:19.132546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.132523    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-6td24"]
Oct 30 18:37:19.139618 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:19.139589    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d8d46c_f388_43a2_9386_26166b60f904.slice/crio-f8c3c830f62ed50e9613361268b486d3392e3f2c599e9b5aece7289cd7f4b3f7 WatchSource:0}: Error finding container f8c3c830f62ed50e9613361268b486d3392e3f2c599e9b5aece7289cd7f4b3f7: Status 404 returned error can't find the container with id f8c3c830f62ed50e9613361268b486d3392e3f2c599e9b5aece7289cd7f4b3f7
Oct 30 18:37:19.246491 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.246444    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl" event={"ID":"72dd3f31-31ed-41a4-a097-8d01f4063960","Type":"ContainerStarted","Data":"003c1ef99612e08e6db1eb054359efcef4d37ee88c008f23add706f4adcdfc52"}
Oct 30 18:37:19.248056 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.248028    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv" event={"ID":"f85b88d9-6748-43f2-a172-8377a7f8e700","Type":"ContainerStarted","Data":"1292b5a7e5daee39aac8ff7ddfb63c6e37a2e0ffe4830a2450ea0e0b3084b05b"}
Oct 30 18:37:19.248214 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.248077    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:19.248214 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.248096    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv" event={"ID":"f85b88d9-6748-43f2-a172-8377a7f8e700","Type":"ContainerStarted","Data":"efa790c6f81707dceaebc4c0d03e73ef0d1b9dffffcb6048ec7c30b239949f0b"}
Oct 30 18:37:19.249277 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.249254    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6td24" event={"ID":"31d8d46c-f388-43a2-9386-26166b60f904","Type":"ContainerStarted","Data":"f8c3c830f62ed50e9613361268b486d3392e3f2c599e9b5aece7289cd7f4b3f7"}
Oct 30 18:37:19.379167 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:19.379099    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv" podStartSLOduration=1.379076746 podStartE2EDuration="1.379076746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:19.376900767 +0000 UTC m=+21.457486992" watchObservedRunningTime="2026-10-30 18:37:19.379076746 +0000 UTC m=+21.459662957"
Oct 30 18:37:20.576181 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.576155    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:20.594248 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.594222    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-778ff5cff-fzqpv"]
Oct 30 18:37:20.597770 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.597750    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.655974 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.655955    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 18:37:20.694279 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.694198    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 18:37:20.694279 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.694218    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 18:37:20.697339 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.697302    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 18:37:20.697443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.697351    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fn5lz\""
Oct 30 18:37:20.697443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.697368    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 18:37:20.704673 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.704647    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-serving-cert\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.704673 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.704670    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-oauth-serving-cert\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.704793 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.704691    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-service-ca\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.704793 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.704710    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-tmp\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.704870 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.704858    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-config\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.704904 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.704880    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwpj\" (UniqueName: \"kubernetes.io/projected/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-kube-api-access-bzwpj\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.704952 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.704933    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-oauth-config\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806025    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-serving-cert\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806059    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-oauth-serving-cert\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806077    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-service-ca\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806291    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-tmp\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806424 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806403    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-config\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806473 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806437    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwpj\" (UniqueName: \"kubernetes.io/projected/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-kube-api-access-bzwpj\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806521 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806477    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-oauth-config\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806755 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806720    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-tmp\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806872 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806777    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-service-ca\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.806932 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806867    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-oauth-serving-cert\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.807032 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.806993    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-config\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.808816 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.808795    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-serving-cert\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.808893 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.808821    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-oauth-config\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.844982 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.844955    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778ff5cff-fzqpv"]
Oct 30 18:37:20.865649 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.865623    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwpj\" (UniqueName: \"kubernetes.io/projected/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-kube-api-access-bzwpj\") pod \"console-778ff5cff-fzqpv\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") " pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.909172 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.909149    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:20.912159 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:20.912122    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:21.260596 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.260563    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl" event={"ID":"72dd3f31-31ed-41a4-a097-8d01f4063960","Type":"ContainerStarted","Data":"4c02f9cd2bf809d903817ba9ab6f953ab72ba3473ffba907ae56a1ad2fa598a3"}
Oct 30 18:37:21.260784 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.260766    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"
Oct 30 18:37:21.265751 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.265711    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl"
Oct 30 18:37:21.300636 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.300491    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778ff5cff-fzqpv"]
Oct 30 18:37:21.302996 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:21.302971    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d58fc6_3f41_40ff_93ed_7cf7ff3c14ce.slice/crio-a561eb1555b54f3a9dfe76f519d15958a1979708265b73488786df7af37585d6 WatchSource:0}: Error finding container a561eb1555b54f3a9dfe76f519d15958a1979708265b73488786df7af37585d6: Status 404 returned error can't find the container with id a561eb1555b54f3a9dfe76f519d15958a1979708265b73488786df7af37585d6
Oct 30 18:37:21.323848 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.323820    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:21.401895 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.401868    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5lk89"
Oct 30 18:37:21.498150 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.498098    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-fhbbl" podStartSLOduration=3.0067603 podStartE2EDuration="4.49808513s" totalImagesPullingTime="1.49132483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:21.380802493 +0000 UTC m=+23.461388700" watchObservedRunningTime="2026-10-30 18:37:21.49808513 +0000 UTC m=+23.578671337"
Oct 30 18:37:21.649146 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:21.649109    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:22.266088 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:22.266055    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff5cff-fzqpv" event={"ID":"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce","Type":"ContainerStarted","Data":"a561eb1555b54f3a9dfe76f519d15958a1979708265b73488786df7af37585d6"}
Oct 30 18:37:22.484770 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:22.484738    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:22.867738 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:22.867687    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:23.057782 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.057747    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"]
Oct 30 18:37:23.061602 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.061578    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.111555 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.111518    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:23.129801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.129730    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb07387c-b9fd-4592-beef-0264e73732d6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.129949 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.129822    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb07387c-b9fd-4592-beef-0264e73732d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.129949 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.129896    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb07387c-b9fd-4592-beef-0264e73732d6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.129949 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.129934    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zdtm\" (UniqueName: \"kubernetes.io/projected/cb07387c-b9fd-4592-beef-0264e73732d6-kube-api-access-7zdtm\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.159721 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.159690    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 18:37:23.159867 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.159834    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 18:37:23.159930 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.159895    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 18:37:23.161060 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.161032    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"]
Oct 30 18:37:23.163467 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.163443    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-znjbz\""
Oct 30 18:37:23.230967 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.230930    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb07387c-b9fd-4592-beef-0264e73732d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.231151 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.230977    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb07387c-b9fd-4592-beef-0264e73732d6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.231151 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.231005    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zdtm\" (UniqueName: \"kubernetes.io/projected/cb07387c-b9fd-4592-beef-0264e73732d6-kube-api-access-7zdtm\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.231151 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.231055    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb07387c-b9fd-4592-beef-0264e73732d6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.231905 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.231883    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cb07387c-b9fd-4592-beef-0264e73732d6-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.233930 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.233903    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cb07387c-b9fd-4592-beef-0264e73732d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.234028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.233996    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cb07387c-b9fd-4592-beef-0264e73732d6-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.295391 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.295355    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zdtm\" (UniqueName: \"kubernetes.io/projected/cb07387c-b9fd-4592-beef-0264e73732d6-kube-api-access-7zdtm\") pod \"prometheus-operator-9cd89b74c-48dr5\" (UID: \"cb07387c-b9fd-4592-beef-0264e73732d6\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.379367 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.379310    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"
Oct 30 18:37:23.500575 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.500533    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:23.771088 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:23.771003    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-48dr5"]
Oct 30 18:37:23.778803 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:23.778773    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb07387c_b9fd_4592_beef_0264e73732d6.slice/crio-2ebb6f581363296781c7f9ace2e3540bdf0a055f8b5a94bc637235ec8414fa38 WatchSource:0}: Error finding container 2ebb6f581363296781c7f9ace2e3540bdf0a055f8b5a94bc637235ec8414fa38: Status 404 returned error can't find the container with id 2ebb6f581363296781c7f9ace2e3540bdf0a055f8b5a94bc637235ec8414fa38
Oct 30 18:37:24.250143 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:24.250107    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:24.278881 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:24.278840    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5" event={"ID":"cb07387c-b9fd-4592-beef-0264e73732d6","Type":"ContainerStarted","Data":"2ebb6f581363296781c7f9ace2e3540bdf0a055f8b5a94bc637235ec8414fa38"}
Oct 30 18:37:24.468791 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:24.468745    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:24.989634 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:24.989597    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:25.442738 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:25.442703    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:25.507031 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:25.506972    5025 patch_prober.go:29] interesting pod/image-registry-b895475ff-2t4mb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:25.507189 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:25.507035    5025 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:25.681461 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:25.681421    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:25.883622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:25.883573    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:26.291889 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:26.291857    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5" event={"ID":"cb07387c-b9fd-4592-beef-0264e73732d6","Type":"ContainerStarted","Data":"332d9a9041607e0a7b39115c9199a170eabaf626a417b02266692a7890f8d0b0"}
Oct 30 18:37:26.293693 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:26.293668    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff5cff-fzqpv" event={"ID":"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce","Type":"ContainerStarted","Data":"61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff"}
Oct 30 18:37:26.431651 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:26.431584    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-778ff5cff-fzqpv" podStartSLOduration=2.4518878060000002 podStartE2EDuration="6.431565756s" totalImagesPullingTime="3.97967795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:26.430345532 +0000 UTC m=+28.510931733" watchObservedRunningTime="2026-10-30 18:37:26.431565756 +0000 UTC m=+28.512151966"
Oct 30 18:37:26.887876 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:26.887841    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:27.215826 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:27.215745    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:27.301051 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:27.301017    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5" event={"ID":"cb07387c-b9fd-4592-beef-0264e73732d6","Type":"ContainerStarted","Data":"f2717ab3ab15f642c6fca3c0f214dc96922d9225283f8eb57649b225ef0c13cb"}
Oct 30 18:37:27.415417 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:27.415356    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-48dr5" podStartSLOduration=2.128923726 podStartE2EDuration="4.415317146s" totalImagesPullingTime="2.28639342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:27.413616636 +0000 UTC m=+29.494202845" watchObservedRunningTime="2026-10-30 18:37:27.415317146 +0000 UTC m=+29.495903356"
Oct 30 18:37:27.495356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:27.495251    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:28.044599 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:28.044564    5025 patch_prober.go:29] interesting pod/image-registry-8d9cdb74b-n2d7w container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:28.045034 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:28.044623    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w" podUID="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:28.087513 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:28.087473    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:37:28.251362 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:28.251310    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gbsgk"
Oct 30 18:37:28.726513 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:28.726476    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:29.898658 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:29.898563    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:30.104906 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.104868    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:30.138775 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.138751    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hktmd"
Oct 30 18:37:30.340801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.340723    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:30.521377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.521344    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7484d6b64c-8s4rr"]
Oct 30 18:37:30.559236 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.559194    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.573039 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.573009    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:30.585578 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.585547    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7484d6b64c-8s4rr"]
Oct 30 18:37:30.698760 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.698727    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-service-ca\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.698947 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.698781    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-oauth-serving-cert\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.698947 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.698865    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-oauth-config\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.698947 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.698932    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-serving-cert\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.699063 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.698959    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-trusted-ca-bundle\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.699063 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.698996    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-console-config\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.699155 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.699078    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmsn\" (UniqueName: \"kubernetes.io/projected/3b867dca-047b-4640-9cde-9d3f1545360b-kube-api-access-jrmsn\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.699155 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.699115    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b867dca-047b-4640-9cde-9d3f1545360b-tmp\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.758148 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.758117    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 18:37:30.800307 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800270    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b867dca-047b-4640-9cde-9d3f1545360b-tmp\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800500 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800371    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-service-ca\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800500 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800401    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-oauth-serving-cert\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800500 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800418    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-oauth-config\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800500 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800468    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-serving-cert\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800500 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800498    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-trusted-ca-bundle\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800771 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800544    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-console-config\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800771 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800615    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmsn\" (UniqueName: \"kubernetes.io/projected/3b867dca-047b-4640-9cde-9d3f1545360b-kube-api-access-jrmsn\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.800771 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.800737    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b867dca-047b-4640-9cde-9d3f1545360b-tmp\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.801204 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.801183    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-service-ca\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.801690 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.801670    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-console-config\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.801793 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.801735    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-oauth-serving-cert\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.802017 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.801992    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-trusted-ca-bundle\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.803448 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.803422    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-serving-cert\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.803809 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.803791    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-oauth-config\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.859586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.859539    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:30.861634 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.861606    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmsn\" (UniqueName: \"kubernetes.io/projected/3b867dca-047b-4640-9cde-9d3f1545360b-kube-api-access-jrmsn\") pod \"console-7484d6b64c-8s4rr\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") " pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.889859 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.889825    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:30.893378 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.893352    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r8nhc"]
Oct 30 18:37:30.917952 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.917917    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:30.917952 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.917957    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:30.918431 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.918048    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:30.918431 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.918109    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:30.923172 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:30.923123    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:37:31.002505 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002473    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-sys\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002657 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002516    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002657 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002619    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-textfile\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002770 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002673    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlkvn\" (UniqueName: \"kubernetes.io/projected/14b90994-57e8-4f52-9751-20a02c662a2c-kube-api-access-wlkvn\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002770 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002702    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-root\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002770 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002760    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b90994-57e8-4f52-9751-20a02c662a2c-metrics-client-ca\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002802    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-tls\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002834    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-wtmp\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.002913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.002858    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.038807 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.038774    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6qrp7\""
Oct 30 18:37:31.038807 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.038774    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 18:37:31.039196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.038988    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 18:37:31.039196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.039094    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 18:37:31.103694 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103665    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlkvn\" (UniqueName: \"kubernetes.io/projected/14b90994-57e8-4f52-9751-20a02c662a2c-kube-api-access-wlkvn\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.103901 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103708    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-root\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.103901 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103758    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b90994-57e8-4f52-9751-20a02c662a2c-metrics-client-ca\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.103901 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103788    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-tls\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.103901 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103812    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-wtmp\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.103901 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103836    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.103901 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103887    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-sys\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103917    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.103979    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-textfile\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.104035    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-wtmp\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.104065    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-root\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.104092    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/14b90994-57e8-4f52-9751-20a02c662a2c-sys\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104467 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.104346    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-textfile\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104697 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.104672    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-accelerators-collector-config\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.104827 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.104676    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14b90994-57e8-4f52-9751-20a02c662a2c-metrics-client-ca\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.106519 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.106498    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-tls\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.106693 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.106664    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/14b90994-57e8-4f52-9751-20a02c662a2c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.108382 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.108357    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:31.165495 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.165461    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlkvn\" (UniqueName: \"kubernetes.io/projected/14b90994-57e8-4f52-9751-20a02c662a2c-kube-api-access-wlkvn\") pod \"node-exporter-r8nhc\" (UID: \"14b90994-57e8-4f52-9751-20a02c662a2c\") " pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.293698 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.293613    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7484d6b64c-8s4rr"]
Oct 30 18:37:31.300307 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:31.300280    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b867dca_047b_4640_9cde_9d3f1545360b.slice/crio-c6b5c56011320c137508cc5d0a902b5490373007a50d33dec9772fc79b9d6c67 WatchSource:0}: Error finding container c6b5c56011320c137508cc5d0a902b5490373007a50d33dec9772fc79b9d6c67: Status 404 returned error can't find the container with id c6b5c56011320c137508cc5d0a902b5490373007a50d33dec9772fc79b9d6c67
Oct 30 18:37:31.304272 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.304253    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r8nhc"
Oct 30 18:37:31.316041 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.316015    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:31.330019 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.329987    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484d6b64c-8s4rr" event={"ID":"3b867dca-047b-4640-9cde-9d3f1545360b","Type":"ContainerStarted","Data":"c6b5c56011320c137508cc5d0a902b5490373007a50d33dec9772fc79b9d6c67"}
Oct 30 18:37:31.336358 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:31.336314    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b90994_57e8_4f52_9751_20a02c662a2c.slice/crio-cacfded9f7f7359b27e9825410728c6334ca6fd6ed9ffcbcb46fb31f0650ea60 WatchSource:0}: Error finding container cacfded9f7f7359b27e9825410728c6334ca6fd6ed9ffcbcb46fb31f0650ea60: Status 404 returned error can't find the container with id cacfded9f7f7359b27e9825410728c6334ca6fd6ed9ffcbcb46fb31f0650ea60
Oct 30 18:37:31.529811 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.529770    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:31.930563 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.930531    5025 patch_prober.go:29] interesting pod/image-registry-b895475ff-2t4mb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:31.931004 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:31.930586    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:32.189645 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:32.189568    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:32.337862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:32.337826    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8nhc" event={"ID":"14b90994-57e8-4f52-9751-20a02c662a2c","Type":"ContainerStarted","Data":"cacfded9f7f7359b27e9825410728c6334ca6fd6ed9ffcbcb46fb31f0650ea60"}
Oct 30 18:37:32.341193 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:32.341137    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484d6b64c-8s4rr" event={"ID":"3b867dca-047b-4640-9cde-9d3f1545360b","Type":"ContainerStarted","Data":"f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299"}
Oct 30 18:37:32.677219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:32.677152    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:33.051977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:33.051897    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:34.353459 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:34.353423    5025 generic.go:356] "Generic (PLEG): container finished" podID="14b90994-57e8-4f52-9751-20a02c662a2c" containerID="444fae98ad99a3c48db8fd7d56221dc654e9c226e66c87e6dde062cbab44bcd0" exitCode=0
Oct 30 18:37:34.353930 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:34.353465    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8nhc" event={"ID":"14b90994-57e8-4f52-9751-20a02c662a2c","Type":"ContainerDied","Data":"444fae98ad99a3c48db8fd7d56221dc654e9c226e66c87e6dde062cbab44bcd0"}
Oct 30 18:37:34.553234 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:34.553176    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7484d6b64c-8s4rr" podStartSLOduration=4.553162286 podStartE2EDuration="4.553162286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:32.461941882 +0000 UTC m=+34.542528090" watchObservedRunningTime="2026-10-30 18:37:34.553162286 +0000 UTC m=+36.633748487"
Oct 30 18:37:35.224187 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.224146    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:35.361529 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.361495    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8nhc" event={"ID":"14b90994-57e8-4f52-9751-20a02c662a2c","Type":"ContainerStarted","Data":"5ae6cefc63f36968f079e90e49ce46b39bc3df1c0f8ae6694d5e53a9a62823c6"}
Oct 30 18:37:35.361957 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.361584    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r8nhc" event={"ID":"14b90994-57e8-4f52-9751-20a02c662a2c","Type":"ContainerStarted","Data":"d427142a68c3000f31b9f174448edf19c1858f139f992602feb3e9e1cc0d2aa8"}
Oct 30 18:37:35.481304 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.481196    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r8nhc" podStartSLOduration=3.447108448 podStartE2EDuration="5.481182745s" totalImagesPullingTime="1.948615269s" totalInitContainerRuntime="85.459028ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:35.479607994 +0000 UTC m=+37.560194201" watchObservedRunningTime="2026-10-30 18:37:35.481182745 +0000 UTC m=+37.561768952"
Oct 30 18:37:35.499928 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.499895    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:35.505964 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.505930    5025 patch_prober.go:29] interesting pod/image-registry-b895475ff-2t4mb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 18:37:35.506096 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.505987    5025 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 18:37:35.723518 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:35.723474    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:36.889574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:36.889534    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:37.097794 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:37.097747    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:37.538022 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:37.537983    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:37.747484 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:37.747454    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:38.042761 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.042726    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:38.380121 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.380083    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-6td24" event={"ID":"31d8d46c-f388-43a2-9386-26166b60f904","Type":"ContainerStarted","Data":"ce7fd3cc089a226a449b346e31d057ce54cb3fcd41be18ca43d3fce75c188b4e"}
Oct 30 18:37:38.380425 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.380373    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:38.388410 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.388383    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7484d6b64c-8s4rr"]
Oct 30 18:37:38.392913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.392892    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-6td24"
Oct 30 18:37:38.431425 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.431391    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7486cd6f74-dggcf"]
Oct 30 18:37:38.436344 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.436295    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.476997 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.476956    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:38.497928 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.497893    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7486cd6f74-dggcf"]
Oct 30 18:37:38.503840 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.503695    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-6td24" podStartSLOduration=2.811715831 podStartE2EDuration="21.5036774s" totalImagesPullingTime="18.691961569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:38.502815958 +0000 UTC m=+40.583402190" watchObservedRunningTime="2026-10-30 18:37:38.5036774 +0000 UTC m=+40.584263610"
Oct 30 18:37:38.580828 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.580791    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-tmp\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.580828 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.580831    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-oauth-serving-cert\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.581073 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.580977    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-service-ca\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.581073 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.581029    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-config\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.581162 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.581109    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-trusted-ca-bundle\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.581214 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.581162    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-oauth-config\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.581214 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.581185    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bvr\" (UniqueName: \"kubernetes.io/projected/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-kube-api-access-w5bvr\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.581371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.581349    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-serving-cert\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682209 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682133    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-trusted-ca-bundle\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682209 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682190    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-oauth-config\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682489 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682213    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bvr\" (UniqueName: \"kubernetes.io/projected/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-kube-api-access-w5bvr\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682489 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682287    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-serving-cert\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682489 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682335    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-tmp\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682489 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682360    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-oauth-serving-cert\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682489 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682403    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-service-ca\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682489 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682441    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-config\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.682886 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.682790    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-tmp\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.683209 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.683181    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-trusted-ca-bundle\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.683395 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.683268    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-config\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.683494 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.683410    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-oauth-serving-cert\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.683807 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.683783    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-service-ca\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.685652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.685627    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-oauth-config\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.687332 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.687291    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-serving-cert\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.741401 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.741352    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:38.746933 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.746907    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bvr\" (UniqueName: \"kubernetes.io/projected/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-kube-api-access-w5bvr\") pod \"console-7486cd6f74-dggcf\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") " pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:38.761891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:38.761613    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:39.170144 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:39.170098    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7486cd6f74-dggcf"]
Oct 30 18:37:39.171148 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:37:39.171095    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d32990_fb9f_4fd1_a4e4_14b35dcd8591.slice/crio-56b48ec4bb9a049ff09e79b4d048cf3f55eb395c40cb12a0f9dfb9945d8952ea WatchSource:0}: Error finding container 56b48ec4bb9a049ff09e79b4d048cf3f55eb395c40cb12a0f9dfb9945d8952ea: Status 404 returned error can't find the container with id 56b48ec4bb9a049ff09e79b4d048cf3f55eb395c40cb12a0f9dfb9945d8952ea
Oct 30 18:37:39.389937 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:39.389807    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7486cd6f74-dggcf" event={"ID":"17d32990-fb9f-4fd1-a4e4-14b35dcd8591","Type":"ContainerStarted","Data":"a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d"}
Oct 30 18:37:39.389937 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:39.389864    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7486cd6f74-dggcf" event={"ID":"17d32990-fb9f-4fd1-a4e4-14b35dcd8591","Type":"ContainerStarted","Data":"56b48ec4bb9a049ff09e79b4d048cf3f55eb395c40cb12a0f9dfb9945d8952ea"}
Oct 30 18:37:39.505864 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:39.505765    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7486cd6f74-dggcf" podStartSLOduration=1.505750954 podStartE2EDuration="1.505750954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:39.505560877 +0000 UTC m=+41.586147123" watchObservedRunningTime="2026-10-30 18:37:39.505750954 +0000 UTC m=+41.586337152"
Oct 30 18:37:39.930634 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:39.930574    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:40.259121 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:40.259044    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-c4cbdd774-cfpcv"
Oct 30 18:37:40.265639 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:40.265604    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:40.483858 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:40.483816    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b895475ff-2t4mb"]
Oct 30 18:37:40.489569 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:40.489504    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:37:40.862283 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:40.862237    5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log" limit=1024
Oct 30 18:37:40.890497 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:40.890470    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:37:43.058588 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.058544    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w" podUID="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" containerName="registry" containerID="cri-o://df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6" gracePeriod=30
Oct 30 18:37:43.356906 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.356877    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:43.412888 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.412849    5025 generic.go:356] "Generic (PLEG): container finished" podID="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" containerID="df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6" exitCode=0
Oct 30 18:37:43.413065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.412908    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w"
Oct 30 18:37:43.413065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.412905    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w" event={"ID":"fbba07e7-11e1-4277-96ff-0a56c96c9ac1","Type":"ContainerDied","Data":"df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6"}
Oct 30 18:37:43.413065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.412967    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d9cdb74b-n2d7w" event={"ID":"fbba07e7-11e1-4277-96ff-0a56c96c9ac1","Type":"ContainerDied","Data":"d8aa3d9905f1a0943cbaac54afbab949087bb89a6604bcd0738d236f4c005c6c"}
Oct 30 18:37:43.413065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.413008    5025 scope.go:122] "RemoveContainer" containerID="df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6"
Oct 30 18:37:43.426667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.426643    5025 scope.go:122] "RemoveContainer" containerID="df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6"
Oct 30 18:37:43.426863 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.426838    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.426953 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.426926    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.427018 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427002    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.427074 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427030    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.427126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427090    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.427188 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427122    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.427188 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427153    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.427282 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427186    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") pod \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\" (UID: \"fbba07e7-11e1-4277-96ff-0a56c96c9ac1\") "
Oct 30 18:37:43.428087 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427649    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:43.428087 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:43.427760    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6\": container with ID starting with df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6 not found: ID does not exist" containerID="df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6"
Oct 30 18:37:43.428087 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.427794    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6"} err="failed to get container status \"df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6\": rpc error: code = NotFound desc = could not find container \"df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6\": container with ID starting with df436fe22948be1d909b42a5bb363eb33eb2d7cf1ff07380e1a468bd710b3ad6 not found: ID does not exist"
Oct 30 18:37:43.428087 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.428023    5025 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-trusted-ca\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.428471 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.428447    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:37:43.431044 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.431009    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:43.431360 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.431309    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:43.431843 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.431785    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:43.432012 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.431981    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "kube-api-access-dvfsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:37:43.432125 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.432010    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:37:43.437039 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.437012    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted" pod "fbba07e7-11e1-4277-96ff-0a56c96c9ac1" (UID: "fbba07e7-11e1-4277-96ff-0a56c96c9ac1"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:37:43.528775 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.528735    5025 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-certificates\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.528775 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.528775    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvfsh\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-kube-api-access-dvfsh\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.529015 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.528790    5025 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-registry-tls\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.529015 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.528801    5025 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-image-registry-private-configuration\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.529015 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.528812    5025 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-installation-pull-secrets\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.529015 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.528825    5025 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-ca-trust-extracted\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.529015 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.528836    5025 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fbba07e7-11e1-4277-96ff-0a56c96c9ac1-bound-sa-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:37:43.838552 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.838515    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8d9cdb74b-n2d7w"]
Oct 30 18:37:43.859728 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:43.859698    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8d9cdb74b-n2d7w"]
Oct 30 18:37:44.059546 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:44.059460    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" path="/var/lib/kubelet/pods/fbba07e7-11e1-4277-96ff-0a56c96c9ac1/volumes"
Oct 30 18:37:48.762847 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:48.762807    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:48.763402 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:48.762879    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:48.768461 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:48.768439    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:49.446409 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:49.446381    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:37:49.669338 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:37:49.669288    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778ff5cff-fzqpv"]
Oct 30 18:37:58.084667 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:37:58.084622    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:38:03.412267 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.412205    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7484d6b64c-8s4rr" podUID="3b867dca-047b-4640-9cde-9d3f1545360b" containerName="console" containerID="cri-o://f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299" gracePeriod=15
Oct 30 18:38:03.668564 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.668508    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:38:03.818223 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818193    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b867dca-047b-4640-9cde-9d3f1545360b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b867dca-047b-4640-9cde-9d3f1545360b-tmp\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818404 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818234    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-oauth-config\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818404 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818289    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-serving-cert\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818404 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818338    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b867dca-047b-4640-9cde-9d3f1545360b-kube-api-access-jrmsn\" (UniqueName: \"kubernetes.io/projected/3b867dca-047b-4640-9cde-9d3f1545360b-kube-api-access-jrmsn\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818454    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-service-ca\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818494    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-trusted-ca-bundle\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818517    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b867dca-047b-4640-9cde-9d3f1545360b-tmp" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:03.818652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818532    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-console-config\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-console-config\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818625    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-oauth-serving-cert\") pod \"3b867dca-047b-4640-9cde-9d3f1545360b\" (UID: \"3b867dca-047b-4640-9cde-9d3f1545360b\") "
Oct 30 18:38:03.818903 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818868    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-service-ca" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:03.818958 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818926    5025 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3b867dca-047b-4640-9cde-9d3f1545360b-tmp\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:03.819032 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.818893    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-trusted-ca-bundle" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:03.819147 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.819080    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-console-config" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:03.819265 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.819244    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-oauth-serving-cert" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:03.820654 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.820629    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b867dca-047b-4640-9cde-9d3f1545360b-kube-api-access-jrmsn" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "kube-api-access-jrmsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:03.820955 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.820933    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-oauth-config" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:03.820955 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.820944    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-serving-cert" pod "3b867dca-047b-4640-9cde-9d3f1545360b" (UID: "3b867dca-047b-4640-9cde-9d3f1545360b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:03.919724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.919652    5025 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-oauth-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:03.919724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.919676    5025 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-oauth-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:03.919724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.919686    5025 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b867dca-047b-4640-9cde-9d3f1545360b-console-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:03.919724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.919695    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrmsn\" (UniqueName: \"kubernetes.io/projected/3b867dca-047b-4640-9cde-9d3f1545360b-kube-api-access-jrmsn\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:03.919724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.919707    5025 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-service-ca\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:03.919724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.919716    5025 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-trusted-ca-bundle\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:03.919724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:03.919724    5025 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b867dca-047b-4640-9cde-9d3f1545360b-console-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:04.513189 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.513154    5025 generic.go:356] "Generic (PLEG): container finished" podID="3b867dca-047b-4640-9cde-9d3f1545360b" containerID="f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299" exitCode=0
Oct 30 18:38:04.513675 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.513200    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7484d6b64c-8s4rr"
Oct 30 18:38:04.513675 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.513213    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484d6b64c-8s4rr" event={"ID":"3b867dca-047b-4640-9cde-9d3f1545360b","Type":"ContainerDied","Data":"f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299"}
Oct 30 18:38:04.513675 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.513251    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7484d6b64c-8s4rr" event={"ID":"3b867dca-047b-4640-9cde-9d3f1545360b","Type":"ContainerDied","Data":"c6b5c56011320c137508cc5d0a902b5490373007a50d33dec9772fc79b9d6c67"}
Oct 30 18:38:04.513675 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.513270    5025 scope.go:122] "RemoveContainer" containerID="f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299"
Oct 30 18:38:04.532219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.532188    5025 scope.go:122] "RemoveContainer" containerID="f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299"
Oct 30 18:38:04.532476 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:38:04.532458    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299\": container with ID starting with f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299 not found: ID does not exist" containerID="f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299"
Oct 30 18:38:04.532539 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.532485    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299"} err="failed to get container status \"f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299\": rpc error: code = NotFound desc = could not find container \"f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299\": container with ID starting with f358e7834b9ff326086dbcf8f19104b27b50cf872c0a48aa20fd1941453f8299 not found: ID does not exist"
Oct 30 18:38:04.702135 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.702106    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7484d6b64c-8s4rr"]
Oct 30 18:38:04.749377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:04.749350    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7484d6b64c-8s4rr"]
Oct 30 18:38:05.507921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.507884    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerName="registry" containerID="cri-o://989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e" gracePeriod=30
Oct 30 18:38:05.787348 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.787266    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:38:05.934513 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934478    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934695 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934524    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934695 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934565    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934821 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934685    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934821 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934753    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934821 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934791    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934838    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934875    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\") pod \"8d964528-47c1-4c62-8185-a27cc4cb8447\" (UID: \"8d964528-47c1-4c62-8185-a27cc4cb8447\") "
Oct 30 18:38:05.934977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.934954    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:05.935185 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.935167    5025 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-trusted-ca\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:05.935745 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.935717    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:05.937440 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.937407    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:05.937440 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.937420    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:05.937622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.937600    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:05.937671 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.937610    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:05.937671 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.937657    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "kube-api-access-5xv4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:05.942115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:05.942092    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted" pod "8d964528-47c1-4c62-8185-a27cc4cb8447" (UID: "8d964528-47c1-4c62-8185-a27cc4cb8447"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:06.035643 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.035612    5025 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-installation-pull-secrets\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:06.035643 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.035642    5025 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-bound-sa-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:06.035843 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.035653    5025 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8d964528-47c1-4c62-8185-a27cc4cb8447-ca-trust-extracted\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:06.035843 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.035662    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xv4l\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-kube-api-access-5xv4l\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:06.035843 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.035671    5025 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-tls\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:06.035843 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.035679    5025 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8d964528-47c1-4c62-8185-a27cc4cb8447-registry-certificates\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:06.035843 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.035689    5025 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8d964528-47c1-4c62-8185-a27cc4cb8447-image-registry-private-configuration\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:06.056290 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.056218    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b867dca-047b-4640-9cde-9d3f1545360b" path="/var/lib/kubelet/pods/3b867dca-047b-4640-9cde-9d3f1545360b/volumes"
Oct 30 18:38:06.524472 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.524442    5025 generic.go:356] "Generic (PLEG): container finished" podID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerID="989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e" exitCode=0
Oct 30 18:38:06.524645 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.524514    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-b895475ff-2t4mb"
Oct 30 18:38:06.524645 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.524531    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" event={"ID":"8d964528-47c1-4c62-8185-a27cc4cb8447","Type":"ContainerDied","Data":"989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e"}
Oct 30 18:38:06.524645 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.524591    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-b895475ff-2t4mb" event={"ID":"8d964528-47c1-4c62-8185-a27cc4cb8447","Type":"ContainerDied","Data":"47a3d724b291dd3512138489acd9f1cd969e323b7eca486ba031685507a0a2bc"}
Oct 30 18:38:06.524645 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.524618    5025 scope.go:122] "RemoveContainer" containerID="989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e"
Oct 30 18:38:06.533254 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.533239    5025 scope.go:122] "RemoveContainer" containerID="989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e"
Oct 30 18:38:06.533520 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:38:06.533500    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e\": container with ID starting with 989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e not found: ID does not exist" containerID="989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e"
Oct 30 18:38:06.533575 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.533527    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e"} err="failed to get container status \"989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e\": rpc error: code = NotFound desc = could not find container \"989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e\": container with ID starting with 989d0f8804911bfeda38aeb51b5ab50816f2bc8b1168fce724e8150dbb17a55e not found: ID does not exist"
Oct 30 18:38:06.749689 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.749658    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-b895475ff-2t4mb"]
Oct 30 18:38:06.797944 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:06.797874    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-b895475ff-2t4mb"]
Oct 30 18:38:08.056065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:08.056033    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" path="/var/lib/kubelet/pods/8d964528-47c1-4c62-8185-a27cc4cb8447/volumes"
Oct 30 18:38:14.561105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.561026    5025 generic.go:356] "Generic (PLEG): container finished" podID="8e96c52d-079e-4b90-b722-b69ad8b12730" containerID="1ef0bd94fb132f3f8d3384d44f6174a71b0d3c6b7ac3fc5a4473162c019e04b0" exitCode=0
Oct 30 18:38:14.561620 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.561103    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mx59k" event={"ID":"8e96c52d-079e-4b90-b722-b69ad8b12730","Type":"ContainerDied","Data":"1ef0bd94fb132f3f8d3384d44f6174a71b0d3c6b7ac3fc5a4473162c019e04b0"}
Oct 30 18:38:14.561620 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.561492    5025 scope.go:122] "RemoveContainer" containerID="1ef0bd94fb132f3f8d3384d44f6174a71b0d3c6b7ac3fc5a4473162c019e04b0"
Oct 30 18:38:14.562796 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.562775    5025 generic.go:356] "Generic (PLEG): container finished" podID="3ee97fd8-b164-40ca-a9ff-e53e0f461d9e" containerID="51f3a2176c0e5fdd3e7bd71645a0b88912d40325f7aa120c15bbd342626ebad7" exitCode=0
Oct 30 18:38:14.562893 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.562857    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb" event={"ID":"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e","Type":"ContainerDied","Data":"51f3a2176c0e5fdd3e7bd71645a0b88912d40325f7aa120c15bbd342626ebad7"}
Oct 30 18:38:14.563139 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.563123    5025 scope.go:122] "RemoveContainer" containerID="51f3a2176c0e5fdd3e7bd71645a0b88912d40325f7aa120c15bbd342626ebad7"
Oct 30 18:38:14.564536 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.564514    5025 generic.go:356] "Generic (PLEG): container finished" podID="c6f92791-c615-4d6e-88e8-1a3f275aa32a" containerID="eb803938a8dfbb92d0a79e163f934fa5a6c2a615174440d97fc22b57eb2f677b" exitCode=0
Oct 30 18:38:14.564641 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.564565    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr" event={"ID":"c6f92791-c615-4d6e-88e8-1a3f275aa32a","Type":"ContainerDied","Data":"eb803938a8dfbb92d0a79e163f934fa5a6c2a615174440d97fc22b57eb2f677b"}
Oct 30 18:38:14.564837 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.564821    5025 scope.go:122] "RemoveContainer" containerID="eb803938a8dfbb92d0a79e163f934fa5a6c2a615174440d97fc22b57eb2f677b"
Oct 30 18:38:14.689898 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:14.689861    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-778ff5cff-fzqpv" podUID="66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" containerName="console" containerID="cri-o://61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff" gracePeriod=15
Oct 30 18:38:15.011122 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.011089    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:38:15.115231 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115152    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-oauth-config\") pod \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") "
Oct 30 18:38:15.115231 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115201    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-serving-cert\") pod \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") "
Oct 30 18:38:15.115446 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115257    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-kube-api-access-bzwpj\" (UniqueName: \"kubernetes.io/projected/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-kube-api-access-bzwpj\") pod \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") "
Oct 30 18:38:15.115446 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115316    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-service-ca\") pod \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") "
Oct 30 18:38:15.115446 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115358    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-config\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-config\") pod \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") "
Oct 30 18:38:15.115446 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115394    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-tmp\") pod \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") "
Oct 30 18:38:15.115610 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115487    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-oauth-serving-cert\") pod \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\" (UID: \"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce\") "
Oct 30 18:38:15.115908 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115885    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-config" pod "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" (UID: "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:15.115908 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.115881    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-service-ca" pod "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" (UID: "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:15.116454 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.116402    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-tmp" pod "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" (UID: "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:15.116729 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.116698    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-oauth-serving-cert" pod "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" (UID: "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:15.118619 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.118592    5025 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-service-ca\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:15.118619 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.118620    5025 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:15.118758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.118634    5025 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-tmp\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:15.118758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.118650    5025 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-oauth-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:15.119019 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.118983    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-oauth-config" pod "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" (UID: "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:15.119274 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.119244    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-kube-api-access-bzwpj" pod "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" (UID: "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce"). InnerVolumeSpecName "kube-api-access-bzwpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:15.119373 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.119278    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-serving-cert" pod "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" (UID: "66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:38:15.219745 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.219669    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzwpj\" (UniqueName: \"kubernetes.io/projected/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-kube-api-access-bzwpj\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:15.219745 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.219704    5025 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-oauth-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:15.219745 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.219719    5025 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce-console-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:15.577493 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.577450    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mx59k" event={"ID":"8e96c52d-079e-4b90-b722-b69ad8b12730","Type":"ContainerStarted","Data":"602b0b72d7e3b2bd78ad417bafccff17fe22f38d0d31889560b3895764a9e384"}
Oct 30 18:38:15.579685 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.579654    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6f9fb" event={"ID":"3ee97fd8-b164-40ca-a9ff-e53e0f461d9e","Type":"ContainerStarted","Data":"6dcbe83ef8e664b520b906e7a16a380e7eee34f397a5570246e7363852f280a5"}
Oct 30 18:38:15.581368 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.581345    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-mgvhr" event={"ID":"c6f92791-c615-4d6e-88e8-1a3f275aa32a","Type":"ContainerStarted","Data":"4c1fd9831f4753bdd05e55b8dbd477f6307dc4d0910e68d94f137b33b389743f"}
Oct 30 18:38:15.582691 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.582669    5025 generic.go:356] "Generic (PLEG): container finished" podID="66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" containerID="61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff" exitCode=0
Oct 30 18:38:15.582789 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.582700    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff5cff-fzqpv" event={"ID":"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce","Type":"ContainerDied","Data":"61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff"}
Oct 30 18:38:15.582789 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.582727    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778ff5cff-fzqpv" event={"ID":"66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce","Type":"ContainerDied","Data":"a561eb1555b54f3a9dfe76f519d15958a1979708265b73488786df7af37585d6"}
Oct 30 18:38:15.582789 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.582724    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778ff5cff-fzqpv"
Oct 30 18:38:15.582789 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.582744    5025 scope.go:122] "RemoveContainer" containerID="61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff"
Oct 30 18:38:15.591029 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.591013    5025 scope.go:122] "RemoveContainer" containerID="61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff"
Oct 30 18:38:15.591274 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:38:15.591256    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff\": container with ID starting with 61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff not found: ID does not exist" containerID="61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff"
Oct 30 18:38:15.591434 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.591280    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff"} err="failed to get container status \"61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff\": rpc error: code = NotFound desc = could not find container \"61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff\": container with ID starting with 61a221161b76d5c52104c1e282f828188f9902acdc1c6d5bc5654e7440975eff not found: ID does not exist"
Oct 30 18:38:15.610523 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.610503    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-69wwp-9m4cv"]
Oct 30 18:38:15.610704 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.610685    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-69wwp-9m4cv" podUID="670b5b50-4c64-404e-b62c-9fe151aa58b2" containerName="insights-gathering" containerID="cri-o://dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5" gracePeriod=30
Oct 30 18:38:15.840892 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.840804    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778ff5cff-fzqpv"]
Oct 30 18:38:15.846569 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.846541    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-778ff5cff-fzqpv"]
Oct 30 18:38:15.940854 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:15.940827    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:38:16.026748 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.026709    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\") pod \"670b5b50-4c64-404e-b62c-9fe151aa58b2\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") "
Oct 30 18:38:16.026924 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.026793    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\" (UniqueName: \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\") pod \"670b5b50-4c64-404e-b62c-9fe151aa58b2\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") "
Oct 30 18:38:16.026924 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.026899    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\") pod \"670b5b50-4c64-404e-b62c-9fe151aa58b2\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") "
Oct 30 18:38:16.027065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.026973    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\") pod \"670b5b50-4c64-404e-b62c-9fe151aa58b2\" (UID: \"670b5b50-4c64-404e-b62c-9fe151aa58b2\") "
Oct 30 18:38:16.027188 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.027095    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path" pod "670b5b50-4c64-404e-b62c-9fe151aa58b2" (UID: "670b5b50-4c64-404e-b62c-9fe151aa58b2"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:38:16.027265 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.027244    5025 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/670b5b50-4c64-404e-b62c-9fe151aa58b2-archives-path\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:16.027340 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.027254    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle" pod "670b5b50-4c64-404e-b62c-9fe151aa58b2" (UID: "670b5b50-4c64-404e-b62c-9fe151aa58b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:16.027436 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.027412    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle" pod "670b5b50-4c64-404e-b62c-9fe151aa58b2" (UID: "670b5b50-4c64-404e-b62c-9fe151aa58b2"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:38:16.029275 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.029247    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j" pod "670b5b50-4c64-404e-b62c-9fe151aa58b2" (UID: "670b5b50-4c64-404e-b62c-9fe151aa58b2"). InnerVolumeSpecName "kube-api-access-r776j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:38:16.055624 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.055600    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" path="/var/lib/kubelet/pods/66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce/volumes"
Oct 30 18:38:16.128966 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.128725    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r776j\" (UniqueName: \"kubernetes.io/projected/670b5b50-4c64-404e-b62c-9fe151aa58b2-kube-api-access-r776j\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:16.128966 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.128757    5025 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-trusted-ca-bundle\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:16.128966 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.128771    5025 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/670b5b50-4c64-404e-b62c-9fe151aa58b2-service-ca-bundle\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:38:16.587990 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.587897    5025 generic.go:356] "Generic (PLEG): container finished" podID="670b5b50-4c64-404e-b62c-9fe151aa58b2" containerID="dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5" exitCode=2
Oct 30 18:38:16.587990 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.587951    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-69wwp-9m4cv"
Oct 30 18:38:16.588486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.587988    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-69wwp-9m4cv" event={"ID":"670b5b50-4c64-404e-b62c-9fe151aa58b2","Type":"ContainerDied","Data":"dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5"}
Oct 30 18:38:16.588486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.588042    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-69wwp-9m4cv" event={"ID":"670b5b50-4c64-404e-b62c-9fe151aa58b2","Type":"ContainerDied","Data":"41d8c76cd0a70c3a84276034efadfa70938977a04dee4d75c3aa0ed1759b89b1"}
Oct 30 18:38:16.588486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.588068    5025 scope.go:122] "RemoveContainer" containerID="dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5"
Oct 30 18:38:16.596487 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.596470    5025 scope.go:122] "RemoveContainer" containerID="dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5"
Oct 30 18:38:16.596744 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:38:16.596728    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5\": container with ID starting with dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5 not found: ID does not exist" containerID="dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5"
Oct 30 18:38:16.596818 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.596750    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5"} err="failed to get container status \"dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5\": rpc error: code = NotFound desc = could not find container \"dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5\": container with ID starting with dee393bd8d364dd359f6b854a90a08cd71f3f1d3c900e4aebd8c3ddd830351c5 not found: ID does not exist"
Oct 30 18:38:16.749019 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.748985    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-69wwp-9m4cv"]
Oct 30 18:38:16.759452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:16.759426    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-69wwp-9m4cv"]
Oct 30 18:38:18.056217 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:18.056187    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="670b5b50-4c64-404e-b62c-9fe151aa58b2" path="/var/lib/kubelet/pods/670b5b50-4c64-404e-b62c-9fe151aa58b2/volumes"
Oct 30 18:38:24.913474 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913444    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-784566c987-d5txq"]
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913772    5025 cpu_manager.go:414] "containerMap: removing container" podUID="670b5b50-4c64-404e-b62c-9fe151aa58b2" containerName="insights-gathering"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913783    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="670b5b50-4c64-404e-b62c-9fe151aa58b2" containerName="insights-gathering"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913797    5025 cpu_manager.go:414] "containerMap: removing container" podUID="3b867dca-047b-4640-9cde-9d3f1545360b" containerName="console"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913802    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b867dca-047b-4640-9cde-9d3f1545360b" containerName="console"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913811    5025 cpu_manager.go:414] "containerMap: removing container" podUID="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" containerName="registry"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913816    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" containerName="registry"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913831    5025 cpu_manager.go:414] "containerMap: removing container" podUID="66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" containerName="console"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913835    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" containerName="console"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913842    5025 cpu_manager.go:414] "containerMap: removing container" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerName="registry"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913847    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerName="registry"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913902    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b867dca-047b-4640-9cde-9d3f1545360b" containerName="console"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913910    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbba07e7-11e1-4277-96ff-0a56c96c9ac1" containerName="registry"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913917    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d58fc6-3f41-40ff-93ed-7cf7ff3c14ce" containerName="console"
Oct 30 18:38:24.913916 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913924    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d964528-47c1-4c62-8185-a27cc4cb8447" containerName="registry"
Oct 30 18:38:24.914338 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.913932    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="670b5b50-4c64-404e-b62c-9fe151aa58b2" containerName="insights-gathering"
Oct 30 18:38:24.916590 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:24.916571    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.019123 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.019092    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784566c987-d5txq"]
Oct 30 18:38:25.102431 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102399    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-oauth-config\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.102607 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102439    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-trusted-ca-bundle\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.102607 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102468    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-serving-cert\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.102607 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102588    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-oauth-serving-cert\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.102741 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102614    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk94c\" (UniqueName: \"kubernetes.io/projected/935a3c1f-5065-488a-8374-c22ee4d14f3c-kube-api-access-wk94c\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.102741 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102645    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-service-ca\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.102741 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102725    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-config\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.102844 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.102768    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935a3c1f-5065-488a-8374-c22ee4d14f3c-tmp\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203251 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203174    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-serving-cert\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203251 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203207    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-oauth-serving-cert\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203510 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203252    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk94c\" (UniqueName: \"kubernetes.io/projected/935a3c1f-5065-488a-8374-c22ee4d14f3c-kube-api-access-wk94c\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203510 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203292    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-service-ca\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203510 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203370    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-config\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203510 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203404    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935a3c1f-5065-488a-8374-c22ee4d14f3c-tmp\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203510 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203476    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-oauth-config\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.203778 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.203529    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-trusted-ca-bundle\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.204140 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.204115    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935a3c1f-5065-488a-8374-c22ee4d14f3c-tmp\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.204239 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.204171    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-service-ca\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.204504 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.204468    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-trusted-ca-bundle\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.204504 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.204488    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-config\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.204639 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.204522    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-oauth-serving-cert\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.205830 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.205813    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-serving-cert\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.206156 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.206139    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-oauth-config\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.259993 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.259969    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk94c\" (UniqueName: \"kubernetes.io/projected/935a3c1f-5065-488a-8374-c22ee4d14f3c-kube-api-access-wk94c\") pod \"console-784566c987-d5txq\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") " pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.527232 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.527156    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:25.899857 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:25.899817    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784566c987-d5txq"]
Oct 30 18:38:25.905887 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:38:25.905860    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935a3c1f_5065_488a_8374_c22ee4d14f3c.slice/crio-a27cefaf1a66048c1945c4181f01371b031ff3fa7a5d760facc2c226f6384a14 WatchSource:0}: Error finding container a27cefaf1a66048c1945c4181f01371b031ff3fa7a5d760facc2c226f6384a14: Status 404 returned error can't find the container with id a27cefaf1a66048c1945c4181f01371b031ff3fa7a5d760facc2c226f6384a14
Oct 30 18:38:26.632580 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:26.632544    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784566c987-d5txq" event={"ID":"935a3c1f-5065-488a-8374-c22ee4d14f3c","Type":"ContainerStarted","Data":"2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee"}
Oct 30 18:38:26.632977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:26.632592    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784566c987-d5txq" event={"ID":"935a3c1f-5065-488a-8374-c22ee4d14f3c","Type":"ContainerStarted","Data":"a27cefaf1a66048c1945c4181f01371b031ff3fa7a5d760facc2c226f6384a14"}
Oct 30 18:38:26.756124 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:26.756071    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-784566c987-d5txq" podStartSLOduration=2.756056701 podStartE2EDuration="2.756056701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:26.755948672 +0000 UTC m=+88.836534906" watchObservedRunningTime="2026-10-30 18:38:26.756056701 +0000 UTC m=+88.836642905"
Oct 30 18:38:28.082412 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:38:28.082376    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:38:35.527400 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:35.527370    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:35.527758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:35.527448    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:35.531974 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:35.531953    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:35.673442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:35.673412    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:38:35.828093 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:38:35.828015    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7486cd6f74-dggcf"]
Oct 30 18:38:58.082660 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:38:58.082626    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:39:00.851812 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:00.851778    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7486cd6f74-dggcf" podUID="17d32990-fb9f-4fd1-a4e4-14b35dcd8591" containerName="console" containerID="cri-o://a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d" gracePeriod=15
Oct 30 18:39:01.103069 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.103013    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:39:01.200746 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200714    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-config\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-config\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.200891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200757    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-tmp\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.200891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200830    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-trusted-ca-bundle\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.200891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200849    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-oauth-config\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.200891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200869    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-serving-cert\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.201099 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200898    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-service-ca\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.201099 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200924    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-kube-api-access-w5bvr\" (UniqueName: \"kubernetes.io/projected/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-kube-api-access-w5bvr\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.201099 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.200964    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-oauth-serving-cert\") pod \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\" (UID: \"17d32990-fb9f-4fd1-a4e4-14b35dcd8591\") "
Oct 30 18:39:01.201099 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.201090    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-tmp" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:39:01.201306 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.201113    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-config" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:01.201306 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.201274    5025 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.201306 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.201289    5025 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-tmp\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.201481 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.201421    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-trusted-ca-bundle" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:01.201519 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.201488    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-oauth-serving-cert" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:01.201519 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.201494    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-service-ca" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:39:01.203245 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.203220    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-oauth-config" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:01.203360 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.203235    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-kube-api-access-w5bvr" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "kube-api-access-w5bvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:39:01.203360 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.203267    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-serving-cert" pod "17d32990-fb9f-4fd1-a4e4-14b35dcd8591" (UID: "17d32990-fb9f-4fd1-a4e4-14b35dcd8591"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:39:01.302170 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.302137    5025 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-trusted-ca-bundle\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.302170 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.302163    5025 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-oauth-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.302170 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.302173    5025 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-console-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.302426 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.302181    5025 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-service-ca\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.302426 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.302191    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5bvr\" (UniqueName: \"kubernetes.io/projected/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-kube-api-access-w5bvr\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.302426 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.302200    5025 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/17d32990-fb9f-4fd1-a4e4-14b35dcd8591-oauth-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:39:01.778105 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.778070    5025 generic.go:356] "Generic (PLEG): container finished" podID="17d32990-fb9f-4fd1-a4e4-14b35dcd8591" containerID="a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d" exitCode=0
Oct 30 18:39:01.778312 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.778144    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7486cd6f74-dggcf"
Oct 30 18:39:01.778312 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.778159    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7486cd6f74-dggcf" event={"ID":"17d32990-fb9f-4fd1-a4e4-14b35dcd8591","Type":"ContainerDied","Data":"a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d"}
Oct 30 18:39:01.778312 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.778235    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7486cd6f74-dggcf" event={"ID":"17d32990-fb9f-4fd1-a4e4-14b35dcd8591","Type":"ContainerDied","Data":"56b48ec4bb9a049ff09e79b4d048cf3f55eb395c40cb12a0f9dfb9945d8952ea"}
Oct 30 18:39:01.778312 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.778257    5025 scope.go:122] "RemoveContainer" containerID="a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d"
Oct 30 18:39:01.787300 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.787284    5025 scope.go:122] "RemoveContainer" containerID="a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d"
Oct 30 18:39:01.787574 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:39:01.787555    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d\": container with ID starting with a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d not found: ID does not exist" containerID="a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d"
Oct 30 18:39:01.787627 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.787581    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d"} err="failed to get container status \"a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d\": rpc error: code = NotFound desc = could not find container \"a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d\": container with ID starting with a0ec46adfd5e0091d06dccc891e19541ee3249d6fbf3f19d2f7ab6a36c52565d not found: ID does not exist"
Oct 30 18:39:01.936611 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.936578    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7486cd6f74-dggcf"]
Oct 30 18:39:01.983682 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:01.983650    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7486cd6f74-dggcf"]
Oct 30 18:39:02.055801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:02.055728    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d32990-fb9f-4fd1-a4e4-14b35dcd8591" path="/var/lib/kubelet/pods/17d32990-fb9f-4fd1-a4e4-14b35dcd8591/volumes"
Oct 30 18:39:28.081001 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:39:28.080922    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:39:42.330258 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.330221    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d554df5d9-6l22r"]
Oct 30 18:39:42.330890 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.330561    5025 cpu_manager.go:414] "containerMap: removing container" podUID="17d32990-fb9f-4fd1-a4e4-14b35dcd8591" containerName="console"
Oct 30 18:39:42.330890 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.330573    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d32990-fb9f-4fd1-a4e4-14b35dcd8591" containerName="console"
Oct 30 18:39:42.330890 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.330639    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d32990-fb9f-4fd1-a4e4-14b35dcd8591" containerName="console"
Oct 30 18:39:42.333558 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.333538    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424058 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424012    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-trusted-ca-bundle\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424058 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424062    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-service-ca\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424291 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424135    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-console-config\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424291 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424173    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-oauth-serving-cert\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424291 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424241    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd1ff139-7441-4227-a047-d46be091c893-console-oauth-config\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424291 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424269    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd1ff139-7441-4227-a047-d46be091c893-tmp\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424533 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424294    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd1ff139-7441-4227-a047-d46be091c893-console-serving-cert\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.424533 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.424309    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxjd7\" (UniqueName: \"kubernetes.io/projected/bd1ff139-7441-4227-a047-d46be091c893-kube-api-access-zxjd7\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.426585 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.426562    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d554df5d9-6l22r"]
Oct 30 18:39:42.524907 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.524875    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-trusted-ca-bundle\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.524907 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.524912    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-service-ca\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.524931    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-console-config\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.524952    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-oauth-serving-cert\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525005    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd1ff139-7441-4227-a047-d46be091c893-console-oauth-config\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525029    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd1ff139-7441-4227-a047-d46be091c893-tmp\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525061    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd1ff139-7441-4227-a047-d46be091c893-console-serving-cert\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525126 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525084    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxjd7\" (UniqueName: \"kubernetes.io/projected/bd1ff139-7441-4227-a047-d46be091c893-kube-api-access-zxjd7\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525757 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525726    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-service-ca\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525757 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525742    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd1ff139-7441-4227-a047-d46be091c893-tmp\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525954 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525870    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-console-config\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525954 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525887    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-trusted-ca-bundle\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.525954 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.525899    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bd1ff139-7441-4227-a047-d46be091c893-oauth-serving-cert\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.527499 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.527477    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bd1ff139-7441-4227-a047-d46be091c893-console-oauth-config\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.533142 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.527702    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bd1ff139-7441-4227-a047-d46be091c893-console-serving-cert\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.581405 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.581314    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxjd7\" (UniqueName: \"kubernetes.io/projected/bd1ff139-7441-4227-a047-d46be091c893-kube-api-access-zxjd7\") pod \"console-5d554df5d9-6l22r\" (UID: \"bd1ff139-7441-4227-a047-d46be091c893\") " pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:42.644801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:42.644772    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:43.017705 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:43.017668    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d554df5d9-6l22r"]
Oct 30 18:39:43.024800 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:39:43.024772    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1ff139_7441_4227_a047_d46be091c893.slice/crio-5a57b7799f3e2b493c6f608ed6694327ba33883db8bc8265dd5c17e84835c038 WatchSource:0}: Error finding container 5a57b7799f3e2b493c6f608ed6694327ba33883db8bc8265dd5c17e84835c038: Status 404 returned error can't find the container with id 5a57b7799f3e2b493c6f608ed6694327ba33883db8bc8265dd5c17e84835c038
Oct 30 18:39:43.945459 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:43.945424    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d554df5d9-6l22r" event={"ID":"bd1ff139-7441-4227-a047-d46be091c893","Type":"ContainerStarted","Data":"31ee39862234d3c74c80aed3d9c8bb7c92698af5d531af677899ca7277997117"}
Oct 30 18:39:43.945826 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:43.945478    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d554df5d9-6l22r" event={"ID":"bd1ff139-7441-4227-a047-d46be091c893","Type":"ContainerStarted","Data":"5a57b7799f3e2b493c6f608ed6694327ba33883db8bc8265dd5c17e84835c038"}
Oct 30 18:39:44.056476 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:44.056426    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d554df5d9-6l22r" podStartSLOduration=2.056412924 podStartE2EDuration="2.056412924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:44.055580898 +0000 UTC m=+166.136167124" watchObservedRunningTime="2026-10-30 18:39:44.056412924 +0000 UTC m=+166.136999177"
Oct 30 18:39:52.645238 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:52.645192    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:52.645692 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:52.645256    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:52.650107 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:52.650085    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:52.985689 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:52.985611    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d554df5d9-6l22r"
Oct 30 18:39:53.135338 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:39:53.135290    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784566c987-d5txq"]
Oct 30 18:39:58.081594 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:39:58.081548    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:40:18.155550 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.155487    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-784566c987-d5txq" podUID="935a3c1f-5065-488a-8374-c22ee4d14f3c" containerName="console" containerID="cri-o://2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee" gracePeriod=15
Oct 30 18:40:18.389569 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.389546    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:40:18.546476 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546401    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/935a3c1f-5065-488a-8374-c22ee4d14f3c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/935a3c1f-5065-488a-8374-c22ee4d14f3c-tmp\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.546476 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546453    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-config\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-config\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.546476 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546477    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-serving-cert\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.546758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546499    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/935a3c1f-5065-488a-8374-c22ee4d14f3c-kube-api-access-wk94c\" (UniqueName: \"kubernetes.io/projected/935a3c1f-5065-488a-8374-c22ee4d14f3c-kube-api-access-wk94c\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.546758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546528    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-service-ca\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-service-ca\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.546758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546653    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-oauth-config\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.546758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546702    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-trusted-ca-bundle\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.546758 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546736    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-oauth-serving-cert\") pod \"935a3c1f-5065-488a-8374-c22ee4d14f3c\" (UID: \"935a3c1f-5065-488a-8374-c22ee4d14f3c\") "
Oct 30 18:40:18.547005 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546754    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935a3c1f-5065-488a-8374-c22ee4d14f3c-tmp" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:40:18.547005 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.546875    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-config" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:18.547104 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547076    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-service-ca" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:18.547149 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547106    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-trusted-ca-bundle" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:18.547149 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547139    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-oauth-serving-cert" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:40:18.547253 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547235    5025 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/935a3c1f-5065-488a-8374-c22ee4d14f3c-tmp\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:18.547305 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547261    5025 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:18.547305 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547276    5025 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-service-ca\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:18.547305 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547293    5025 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-trusted-ca-bundle\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:18.547471 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.547306    5025 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/935a3c1f-5065-488a-8374-c22ee4d14f3c-oauth-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:18.548829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.548796    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-oauth-config" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:40:18.548933 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.548848    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935a3c1f-5065-488a-8374-c22ee4d14f3c-kube-api-access-wk94c" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "kube-api-access-wk94c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:40:18.548983 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.548935    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-serving-cert" pod "935a3c1f-5065-488a-8374-c22ee4d14f3c" (UID: "935a3c1f-5065-488a-8374-c22ee4d14f3c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:40:18.648560 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.648529    5025 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-serving-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:18.648560 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.648555    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk94c\" (UniqueName: \"kubernetes.io/projected/935a3c1f-5065-488a-8374-c22ee4d14f3c-kube-api-access-wk94c\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:18.648748 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:18.648568    5025 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/935a3c1f-5065-488a-8374-c22ee4d14f3c-console-oauth-config\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:40:19.084356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.084299    5025 generic.go:356] "Generic (PLEG): container finished" podID="935a3c1f-5065-488a-8374-c22ee4d14f3c" containerID="2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee" exitCode=0
Oct 30 18:40:19.084356 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.084361    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784566c987-d5txq"
Oct 30 18:40:19.084563 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.084359    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784566c987-d5txq" event={"ID":"935a3c1f-5065-488a-8374-c22ee4d14f3c","Type":"ContainerDied","Data":"2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee"}
Oct 30 18:40:19.084563 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.084414    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784566c987-d5txq" event={"ID":"935a3c1f-5065-488a-8374-c22ee4d14f3c","Type":"ContainerDied","Data":"a27cefaf1a66048c1945c4181f01371b031ff3fa7a5d760facc2c226f6384a14"}
Oct 30 18:40:19.084563 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.084435    5025 scope.go:122] "RemoveContainer" containerID="2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee"
Oct 30 18:40:19.092907 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.092891    5025 scope.go:122] "RemoveContainer" containerID="2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee"
Oct 30 18:40:19.093174 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:40:19.093152    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee\": container with ID starting with 2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee not found: ID does not exist" containerID="2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee"
Oct 30 18:40:19.093224 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.093183    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee"} err="failed to get container status \"2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee\": rpc error: code = NotFound desc = could not find container \"2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee\": container with ID starting with 2db3940e482490141a0f6a0a7b6be68f71e59816faa8fb0615c502d9f501d5ee not found: ID does not exist"
Oct 30 18:40:19.243735 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.243710    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784566c987-d5txq"]
Oct 30 18:40:19.290751 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:19.290722    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-784566c987-d5txq"]
Oct 30 18:40:20.056089 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:40:20.056058    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="935a3c1f-5065-488a-8374-c22ee4d14f3c" path="/var/lib/kubelet/pods/935a3c1f-5065-488a-8374-c22ee4d14f3c/volumes"
Oct 30 18:40:28.081270 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:40:28.081239    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:40:58.081932 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:40:58.081896    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:41:28.081143 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:41:28.081105    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:41:58.088503 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:41:58.088472    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:41:58.089391 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:41:58.089373    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:41:58.089566 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:41:58.089547    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:42:28.081414 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:42:28.081341    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:42:58.082032 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:42:58.081994    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:43:28.081201 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:43:28.081166    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:43:58.081584 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:43:58.081547    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:44:28.080658 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:44:28.080622    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:44:54.643767 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.643737    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"]
Oct 30 18:44:54.644173 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.644053    5025 cpu_manager.go:414] "containerMap: removing container" podUID="935a3c1f-5065-488a-8374-c22ee4d14f3c" containerName="console"
Oct 30 18:44:54.644173 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.644064    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="935a3c1f-5065-488a-8374-c22ee4d14f3c" containerName="console"
Oct 30 18:44:54.644173 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.644134    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="935a3c1f-5065-488a-8374-c22ee4d14f3c" containerName="console"
Oct 30 18:44:54.647108 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.647088    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.719256 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.719220    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"]
Oct 30 18:44:54.728977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.728952    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e487ea-7514-4a96-b4a4-ea2acf81e31f-cert\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.729113 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.729003    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/11e487ea-7514-4a96-b4a4-ea2acf81e31f-manager-config\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.729113 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.729088    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/11e487ea-7514-4a96-b4a4-ea2acf81e31f-metrics-cert\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.729254 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.729225    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnps\" (UniqueName: \"kubernetes.io/projected/11e487ea-7514-4a96-b4a4-ea2acf81e31f-kube-api-access-dmnps\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.779791 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.779767    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 18:44:54.802911 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.802884    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 18:44:54.802911 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.802886    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 18:44:54.803069 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.802923    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 18:44:54.803069 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.802943    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-ckbd9\""
Oct 30 18:44:54.803069 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.803022    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 18:44:54.830512 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.830485    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnps\" (UniqueName: \"kubernetes.io/projected/11e487ea-7514-4a96-b4a4-ea2acf81e31f-kube-api-access-dmnps\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.830617 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.830547    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e487ea-7514-4a96-b4a4-ea2acf81e31f-cert\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.830617 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.830596    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/11e487ea-7514-4a96-b4a4-ea2acf81e31f-manager-config\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.830734 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.830633    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/11e487ea-7514-4a96-b4a4-ea2acf81e31f-metrics-cert\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.831237 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.831219    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/11e487ea-7514-4a96-b4a4-ea2acf81e31f-manager-config\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.832910 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.832890    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/11e487ea-7514-4a96-b4a4-ea2acf81e31f-cert\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.833024 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.833008    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/11e487ea-7514-4a96-b4a4-ea2acf81e31f-metrics-cert\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.889134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.889112    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnps\" (UniqueName: \"kubernetes.io/projected/11e487ea-7514-4a96-b4a4-ea2acf81e31f-kube-api-access-dmnps\") pod \"lws-controller-manager-d5d6bc6bf-tmqvp\" (UID: \"11e487ea-7514-4a96-b4a4-ea2acf81e31f\") " pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:54.959813 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:54.959756    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:44:55.330223 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:55.330153    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"]
Oct 30 18:44:55.336534 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:44:55.336507    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e487ea_7514_4a96_b4a4_ea2acf81e31f.slice/crio-5fe851fda859fe4d15a7ede84d30d5a355d8d5ce162066aa03473914998c83a2 WatchSource:0}: Error finding container 5fe851fda859fe4d15a7ede84d30d5a355d8d5ce162066aa03473914998c83a2: Status 404 returned error can't find the container with id 5fe851fda859fe4d15a7ede84d30d5a355d8d5ce162066aa03473914998c83a2
Oct 30 18:44:55.338503 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:55.338486    5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:44:56.175351 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:44:56.175302    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp" event={"ID":"11e487ea-7514-4a96-b4a4-ea2acf81e31f","Type":"ContainerStarted","Data":"5fe851fda859fe4d15a7ede84d30d5a355d8d5ce162066aa03473914998c83a2"}
Oct 30 18:44:58.088141 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:44:58.088105    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:45:00.193025 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:00.192985    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp" event={"ID":"11e487ea-7514-4a96-b4a4-ea2acf81e31f","Type":"ContainerStarted","Data":"13b6d5f98b047ca3a8628199f5578c972fa9a80bd35370a5fcc81f1df9a517ce"}
Oct 30 18:45:00.193466 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:00.193045    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:45:00.304943 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:00.304889    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp" podStartSLOduration=2.441827268 podStartE2EDuration="6.304875086s" totalImagesPullingTime="3.863047818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:00.304165377 +0000 UTC m=+482.384751587" watchObservedRunningTime="2026-10-30 18:45:00.304875086 +0000 UTC m=+482.385461358"
Oct 30 18:45:11.199483 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:11.199401    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d5d6bc6bf-tmqvp"
Oct 30 18:45:28.083760 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:45:28.083722    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:45:33.295559 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.295523    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"]
Oct 30 18:45:33.302271 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.302247    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"
Oct 30 18:45:33.360159 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.360128    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 18:45:33.400571 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.400540    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 18:45:33.400723 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.400576    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 18:45:33.400829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.400812    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5stgk\""
Oct 30 18:45:33.402033 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.402015    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"]
Oct 30 18:45:33.465430 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.465402    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m497v\" (UniqueName: \"kubernetes.io/projected/e3398bb6-dec1-4048-8be9-11b71e740591-kube-api-access-m497v\") pod \"dns-operator-controller-manager-68df894cb4-2b7bx\" (UID: \"e3398bb6-dec1-4048-8be9-11b71e740591\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"
Oct 30 18:45:33.566052 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.565977    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m497v\" (UniqueName: \"kubernetes.io/projected/e3398bb6-dec1-4048-8be9-11b71e740591-kube-api-access-m497v\") pod \"dns-operator-controller-manager-68df894cb4-2b7bx\" (UID: \"e3398bb6-dec1-4048-8be9-11b71e740591\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"
Oct 30 18:45:33.622828 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.622794    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m497v\" (UniqueName: \"kubernetes.io/projected/e3398bb6-dec1-4048-8be9-11b71e740591-kube-api-access-m497v\") pod \"dns-operator-controller-manager-68df894cb4-2b7bx\" (UID: \"e3398bb6-dec1-4048-8be9-11b71e740591\") " pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"
Oct 30 18:45:33.914953 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:33.914915    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"
Oct 30 18:45:34.368585 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:34.368553    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"]
Oct 30 18:45:34.374699 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:45:34.374669    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3398bb6_dec1_4048_8be9_11b71e740591.slice/crio-bda4c8860d9432109cdacb2fe509332c92fdb3c99fd7e8f7ed0eaa41bdaf4926 WatchSource:0}: Error finding container bda4c8860d9432109cdacb2fe509332c92fdb3c99fd7e8f7ed0eaa41bdaf4926: Status 404 returned error can't find the container with id bda4c8860d9432109cdacb2fe509332c92fdb3c99fd7e8f7ed0eaa41bdaf4926
Oct 30 18:45:35.333277 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:35.333238    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx" event={"ID":"e3398bb6-dec1-4048-8be9-11b71e740591","Type":"ContainerStarted","Data":"bda4c8860d9432109cdacb2fe509332c92fdb3c99fd7e8f7ed0eaa41bdaf4926"}
Oct 30 18:45:37.104063 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.104026    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"]
Oct 30 18:45:37.113448 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.113423    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.160492 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.160459    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"]
Oct 30 18:45:37.196223 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.196188    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f4a57e2-d306-4708-92d8-8d018614ebfc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.196414 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.196230    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f4a57e2-d306-4708-92d8-8d018614ebfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.196414 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.196253    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlhk4\" (UniqueName: \"kubernetes.io/projected/0f4a57e2-d306-4708-92d8-8d018614ebfc-kube-api-access-vlhk4\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.206109 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.206074    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 18:45:37.206109 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.206094    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2s4f7\""
Oct 30 18:45:37.206109 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.206084    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 18:45:37.297261 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.297218    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f4a57e2-d306-4708-92d8-8d018614ebfc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.297471 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.297267    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f4a57e2-d306-4708-92d8-8d018614ebfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.297471 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.297297    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlhk4\" (UniqueName: \"kubernetes.io/projected/0f4a57e2-d306-4708-92d8-8d018614ebfc-kube-api-access-vlhk4\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.297768 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:45:37.297738    5025 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Oct 30 18:45:37.297891 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:45:37.297833    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4a57e2-d306-4708-92d8-8d018614ebfc-plugin-serving-cert podName:0f4a57e2-d306-4708-92d8-8d018614ebfc nodeName:}" failed. No retries permitted until 2026-10-30 18:45:37.79781526 +0000 UTC m=+519.878401458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0f4a57e2-d306-4708-92d8-8d018614ebfc-plugin-serving-cert") pod "kuadrant-console-plugin-66c4fb575d-6h94n" (UID: "0f4a57e2-d306-4708-92d8-8d018614ebfc") : secret "plugin-serving-cert" not found
Oct 30 18:45:37.297958 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.297943    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f4a57e2-d306-4708-92d8-8d018614ebfc-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.355395 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.355279    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlhk4\" (UniqueName: \"kubernetes.io/projected/0f4a57e2-d306-4708-92d8-8d018614ebfc-kube-api-access-vlhk4\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.802036 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.802002    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f4a57e2-d306-4708-92d8-8d018614ebfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:37.804331 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:37.804293    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0f4a57e2-d306-4708-92d8-8d018614ebfc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6h94n\" (UID: \"0f4a57e2-d306-4708-92d8-8d018614ebfc\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:38.035901 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:38.035867    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"
Oct 30 18:45:38.352178 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:38.352088    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx" event={"ID":"e3398bb6-dec1-4048-8be9-11b71e740591","Type":"ContainerStarted","Data":"c2df439225fffe0de18f5832d71da5d25de099b94e9672692353fe17c746116f"}
Oct 30 18:45:38.352652 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:38.352235    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"
Oct 30 18:45:38.410437 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:45:38.410410    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f4a57e2_d306_4708_92d8_8d018614ebfc.slice/crio-6299403809bbe9388836bc6d049543a866c6eca28912fc522f3d7dcc8b6df94b WatchSource:0}: Error finding container 6299403809bbe9388836bc6d049543a866c6eca28912fc522f3d7dcc8b6df94b: Status 404 returned error can't find the container with id 6299403809bbe9388836bc6d049543a866c6eca28912fc522f3d7dcc8b6df94b
Oct 30 18:45:38.451069 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:38.451031    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n"]
Oct 30 18:45:38.498173 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:38.498123    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx" podStartSLOduration=2.047115876 podStartE2EDuration="5.498107882s" totalImagesPullingTime="3.450992006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:38.462520282 +0000 UTC m=+520.543106512" watchObservedRunningTime="2026-10-30 18:45:38.498107882 +0000 UTC m=+520.578694089"
Oct 30 18:45:39.357268 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:39.357229    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n" event={"ID":"0f4a57e2-d306-4708-92d8-8d018614ebfc","Type":"ContainerStarted","Data":"6299403809bbe9388836bc6d049543a866c6eca28912fc522f3d7dcc8b6df94b"}
Oct 30 18:45:44.381270 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:44.381234    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n" event={"ID":"0f4a57e2-d306-4708-92d8-8d018614ebfc","Type":"ContainerStarted","Data":"372e4dab0234d53a4ed5aacd805f50ae5570694c11ca6563811896378e32ece3"}
Oct 30 18:45:44.494177 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:44.494130    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6h94n" podStartSLOduration=2.173132394 podStartE2EDuration="7.494117831s" totalImagesPullingTime="5.320985437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:44.493596941 +0000 UTC m=+526.574183149" watchObservedRunningTime="2026-10-30 18:45:44.494117831 +0000 UTC m=+526.574704038"
Oct 30 18:45:49.360230 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:45:49.360202    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-68df894cb4-2b7bx"
Oct 30 18:45:58.081479 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:45:58.081442    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:46:28.081988 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:46:28.081885    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:46:34.900926 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:34.900888    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"]
Oct 30 18:46:34.904382 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:34.904365    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:34.957161 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:34.957138    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 18:46:34.957938 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:34.957914    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"]
Oct 30 18:46:35.067393 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.067361    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h656d\" (UniqueName: \"kubernetes.io/projected/cac5c5f3-255a-4afe-aa56-0744d9853db9-kube-api-access-h656d\") pod \"limitador-limitador-74dfcfb894-vdfqs\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:35.067586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.067420    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cac5c5f3-255a-4afe-aa56-0744d9853db9-config-file\") pod \"limitador-limitador-74dfcfb894-vdfqs\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:35.139540 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.139508    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"]
Oct 30 18:46:35.168532 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.168477    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cac5c5f3-255a-4afe-aa56-0744d9853db9-config-file\") pod \"limitador-limitador-74dfcfb894-vdfqs\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:35.168626 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.168558    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h656d\" (UniqueName: \"kubernetes.io/projected/cac5c5f3-255a-4afe-aa56-0744d9853db9-kube-api-access-h656d\") pod \"limitador-limitador-74dfcfb894-vdfqs\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:35.169097 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.169076    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cac5c5f3-255a-4afe-aa56-0744d9853db9-config-file\") pod \"limitador-limitador-74dfcfb894-vdfqs\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:35.224075 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.224045    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h656d\" (UniqueName: \"kubernetes.io/projected/cac5c5f3-255a-4afe-aa56-0744d9853db9-kube-api-access-h656d\") pod \"limitador-limitador-74dfcfb894-vdfqs\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:35.517215 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.517120    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:35.883451 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:35.883419    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"]
Oct 30 18:46:35.889422 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:46:35.889385    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac5c5f3_255a_4afe_aa56_0744d9853db9.slice/crio-d788700548b239b3c3d9c7cbafce6f4707508697d230363171b8d1b528acdb38 WatchSource:0}: Error finding container d788700548b239b3c3d9c7cbafce6f4707508697d230363171b8d1b528acdb38: Status 404 returned error can't find the container with id d788700548b239b3c3d9c7cbafce6f4707508697d230363171b8d1b528acdb38
Oct 30 18:46:36.607563 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:36.607525    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs" event={"ID":"cac5c5f3-255a-4afe-aa56-0744d9853db9","Type":"ContainerStarted","Data":"d788700548b239b3c3d9c7cbafce6f4707508697d230363171b8d1b528acdb38"}
Oct 30 18:46:38.616970 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:38.616925    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs" event={"ID":"cac5c5f3-255a-4afe-aa56-0744d9853db9","Type":"ContainerStarted","Data":"96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe"}
Oct 30 18:46:38.617399 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:38.617024    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:38.729692 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:38.729640    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs" podStartSLOduration=2.547163382 podStartE2EDuration="4.729627843s" totalImagesPullingTime="2.182464461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:38.728048846 +0000 UTC m=+580.808635054" watchObservedRunningTime="2026-10-30 18:46:38.729627843 +0000 UTC m=+580.810214052"
Oct 30 18:46:49.622615 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:49.622585    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:53.319220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:53.319179    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"]
Oct 30 18:46:53.319648 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:53.319378    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs" podUID="cac5c5f3-255a-4afe-aa56-0744d9853db9" containerName="limitador" containerID="cri-o://96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe" gracePeriod=30
Oct 30 18:46:53.864425 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:53.864401    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:53.925178 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:53.925118    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac5c5f3-255a-4afe-aa56-0744d9853db9-config-file\" (UniqueName: \"kubernetes.io/configmap/cac5c5f3-255a-4afe-aa56-0744d9853db9-config-file\") pod \"cac5c5f3-255a-4afe-aa56-0744d9853db9\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") "
Oct 30 18:46:53.925178 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:53.925169    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac5c5f3-255a-4afe-aa56-0744d9853db9-kube-api-access-h656d\" (UniqueName: \"kubernetes.io/projected/cac5c5f3-255a-4afe-aa56-0744d9853db9-kube-api-access-h656d\") pod \"cac5c5f3-255a-4afe-aa56-0744d9853db9\" (UID: \"cac5c5f3-255a-4afe-aa56-0744d9853db9\") "
Oct 30 18:46:53.925491 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:53.925468    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac5c5f3-255a-4afe-aa56-0744d9853db9-config-file" pod "cac5c5f3-255a-4afe-aa56-0744d9853db9" (UID: "cac5c5f3-255a-4afe-aa56-0744d9853db9"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:46:53.927219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:53.927193    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac5c5f3-255a-4afe-aa56-0744d9853db9-kube-api-access-h656d" pod "cac5c5f3-255a-4afe-aa56-0744d9853db9" (UID: "cac5c5f3-255a-4afe-aa56-0744d9853db9"). InnerVolumeSpecName "kube-api-access-h656d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:46:54.026488 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.026459    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h656d\" (UniqueName: \"kubernetes.io/projected/cac5c5f3-255a-4afe-aa56-0744d9853db9-kube-api-access-h656d\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:54.026488 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.026485    5025 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cac5c5f3-255a-4afe-aa56-0744d9853db9-config-file\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:46:54.683919 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.683881    5025 generic.go:356] "Generic (PLEG): container finished" podID="cac5c5f3-255a-4afe-aa56-0744d9853db9" containerID="96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe" exitCode=0
Oct 30 18:46:54.684408 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.683936    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"
Oct 30 18:46:54.684408 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.683945    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs" event={"ID":"cac5c5f3-255a-4afe-aa56-0744d9853db9","Type":"ContainerDied","Data":"96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe"}
Oct 30 18:46:54.684408 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.684029    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-vdfqs" event={"ID":"cac5c5f3-255a-4afe-aa56-0744d9853db9","Type":"ContainerDied","Data":"d788700548b239b3c3d9c7cbafce6f4707508697d230363171b8d1b528acdb38"}
Oct 30 18:46:54.684408 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.684062    5025 scope.go:122] "RemoveContainer" containerID="96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe"
Oct 30 18:46:54.692414 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.692398    5025 scope.go:122] "RemoveContainer" containerID="96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe"
Oct 30 18:46:54.692697 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:46:54.692678    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe\": container with ID starting with 96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe not found: ID does not exist" containerID="96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe"
Oct 30 18:46:54.692757 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.692709    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe"} err="failed to get container status \"96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe\": rpc error: code = NotFound desc = could not find container \"96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe\": container with ID starting with 96b7a9a3a31ddb5c20c6da3787064a4b708c23fa1952b87cb2fc2ef84c356dfe not found: ID does not exist"
Oct 30 18:46:54.842551 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.842522    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"]
Oct 30 18:46:54.889778 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:54.889753    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-vdfqs"]
Oct 30 18:46:56.058444 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:56.058411    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac5c5f3-255a-4afe-aa56-0744d9853db9" path="/var/lib/kubelet/pods/cac5c5f3-255a-4afe-aa56-0744d9853db9/volumes"
Oct 30 18:46:58.081198 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:46:58.081164    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:46:58.115195 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:58.115167    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:46:58.115928 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:46:58.115909    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:47:28.081486 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:47:28.081445    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:47:42.843829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.843799    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-bpktl"]
Oct 30 18:47:42.844242 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.844132    5025 cpu_manager.go:414] "containerMap: removing container" podUID="cac5c5f3-255a-4afe-aa56-0744d9853db9" containerName="limitador"
Oct 30 18:47:42.844242 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.844143    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac5c5f3-255a-4afe-aa56-0744d9853db9" containerName="limitador"
Oct 30 18:47:42.844242 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.844208    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac5c5f3-255a-4afe-aa56-0744d9853db9" containerName="limitador"
Oct 30 18:47:42.847296 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.847266    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:42.898331 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.898292    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 18:47:42.944768 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.944737    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Oct 30 18:47:42.944938 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.944834    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 18:47:42.944938 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.944900    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-hl94m\""
Oct 30 18:47:42.946486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.946459    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-bpktl"]
Oct 30 18:47:42.947501 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.947480    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-9vh6s"]
Oct 30 18:47:42.947791 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.947770    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkqkk\" (UniqueName: \"kubernetes.io/projected/6ba78344-8e28-4aca-8240-878d9749c7e5-kube-api-access-dkqkk\") pod \"kserve-controller-manager-84d86bb774-bpktl\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") " pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:42.947877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.947819    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ba78344-8e28-4aca-8240-878d9749c7e5-cert\") pod \"kserve-controller-manager-84d86bb774-bpktl\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") " pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:42.950705 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.950687    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:42.996576 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:42.996548    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-9vh6s"]
Oct 30 18:47:43.002092 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.002068    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Oct 30 18:47:43.002290 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.002271    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5xchp\""
Oct 30 18:47:43.048476 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.048448    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ba78344-8e28-4aca-8240-878d9749c7e5-cert\") pod \"kserve-controller-manager-84d86bb774-bpktl\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") " pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:43.048602 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.048494    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcg45\" (UniqueName: \"kubernetes.io/projected/4af92a32-5302-4a66-8a6c-d49beb13374b-kube-api-access-rcg45\") pod \"seaweedfs-78bb9c6ff8-9vh6s\" (UID: \"4af92a32-5302-4a66-8a6c-d49beb13374b\") " pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:43.048602 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.048544    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkqkk\" (UniqueName: \"kubernetes.io/projected/6ba78344-8e28-4aca-8240-878d9749c7e5-kube-api-access-dkqkk\") pod \"kserve-controller-manager-84d86bb774-bpktl\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") " pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:43.048602 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.048564    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4af92a32-5302-4a66-8a6c-d49beb13374b-data\") pod \"seaweedfs-78bb9c6ff8-9vh6s\" (UID: \"4af92a32-5302-4a66-8a6c-d49beb13374b\") " pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:43.050927 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.050909    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ba78344-8e28-4aca-8240-878d9749c7e5-cert\") pod \"kserve-controller-manager-84d86bb774-bpktl\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") " pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:43.126775 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.126753    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkqkk\" (UniqueName: \"kubernetes.io/projected/6ba78344-8e28-4aca-8240-878d9749c7e5-kube-api-access-dkqkk\") pod \"kserve-controller-manager-84d86bb774-bpktl\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") " pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:43.149347 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.149306    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcg45\" (UniqueName: \"kubernetes.io/projected/4af92a32-5302-4a66-8a6c-d49beb13374b-kube-api-access-rcg45\") pod \"seaweedfs-78bb9c6ff8-9vh6s\" (UID: \"4af92a32-5302-4a66-8a6c-d49beb13374b\") " pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:43.149482 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.149401    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4af92a32-5302-4a66-8a6c-d49beb13374b-data\") pod \"seaweedfs-78bb9c6ff8-9vh6s\" (UID: \"4af92a32-5302-4a66-8a6c-d49beb13374b\") " pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:43.149732 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.149713    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4af92a32-5302-4a66-8a6c-d49beb13374b-data\") pod \"seaweedfs-78bb9c6ff8-9vh6s\" (UID: \"4af92a32-5302-4a66-8a6c-d49beb13374b\") " pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:43.161274 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.161257    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:43.214804 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.214773    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcg45\" (UniqueName: \"kubernetes.io/projected/4af92a32-5302-4a66-8a6c-d49beb13374b-kube-api-access-rcg45\") pod \"seaweedfs-78bb9c6ff8-9vh6s\" (UID: \"4af92a32-5302-4a66-8a6c-d49beb13374b\") " pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:43.264997 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.264968    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:43.527908 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.527840    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-bpktl"]
Oct 30 18:47:43.533542 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:47:43.533516    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba78344_8e28_4aca_8240_878d9749c7e5.slice/crio-6863b34d2664cd44f61426d1fb90e13d3e7afcd07bdb1c1e7569d92f93ff5cd0 WatchSource:0}: Error finding container 6863b34d2664cd44f61426d1fb90e13d3e7afcd07bdb1c1e7569d92f93ff5cd0: Status 404 returned error can't find the container with id 6863b34d2664cd44f61426d1fb90e13d3e7afcd07bdb1c1e7569d92f93ff5cd0
Oct 30 18:47:43.642991 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.642961    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-78bb9c6ff8-9vh6s"]
Oct 30 18:47:43.649164 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:47:43.649138    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af92a32_5302_4a66_8a6c_d49beb13374b.slice/crio-6b2abb02f139c697f5fadacf0c97572a2f3e74643e8ee71596116ee8a2ad307d WatchSource:0}: Error finding container 6b2abb02f139c697f5fadacf0c97572a2f3e74643e8ee71596116ee8a2ad307d: Status 404 returned error can't find the container with id 6b2abb02f139c697f5fadacf0c97572a2f3e74643e8ee71596116ee8a2ad307d
Oct 30 18:47:43.900093 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.900058    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-9vh6s" event={"ID":"4af92a32-5302-4a66-8a6c-d49beb13374b","Type":"ContainerStarted","Data":"6b2abb02f139c697f5fadacf0c97572a2f3e74643e8ee71596116ee8a2ad307d"}
Oct 30 18:47:43.901219 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:43.901198    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84d86bb774-bpktl" event={"ID":"6ba78344-8e28-4aca-8240-878d9749c7e5","Type":"ContainerStarted","Data":"6863b34d2664cd44f61426d1fb90e13d3e7afcd07bdb1c1e7569d92f93ff5cd0"}
Oct 30 18:47:46.917860 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:46.917820    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84d86bb774-bpktl" event={"ID":"6ba78344-8e28-4aca-8240-878d9749c7e5","Type":"ContainerStarted","Data":"b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e"}
Oct 30 18:47:46.918274 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:46.917938    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:47:47.029201 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:47.029142    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-84d86bb774-bpktl" podStartSLOduration=1.75447968 podStartE2EDuration="5.029122173s" totalImagesPullingTime="3.274642493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:47.028170022 +0000 UTC m=+649.108756246" watchObservedRunningTime="2026-10-30 18:47:47.029122173 +0000 UTC m=+649.109708381"
Oct 30 18:47:48.929786 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:48.929754    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-78bb9c6ff8-9vh6s" event={"ID":"4af92a32-5302-4a66-8a6c-d49beb13374b","Type":"ContainerStarted","Data":"2dd2952f0d4c5ada90f1438d8ee50794e4b8e5fc4bca93f660f2babe60b31681"}
Oct 30 18:47:48.930185 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:48.929862    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:49.040168 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:49.040117    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-78bb9c6ff8-9vh6s" podStartSLOduration=2.290161536 podStartE2EDuration="7.040102792s" totalImagesPullingTime="4.749941256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:49.039180618 +0000 UTC m=+651.119766827" watchObservedRunningTime="2026-10-30 18:47:49.040102792 +0000 UTC m=+651.120688999"
Oct 30 18:47:54.935499 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:47:54.935468    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-78bb9c6ff8-9vh6s"
Oct 30 18:47:58.081516 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:47:58.081482    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:48:17.930092 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:17.930020    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:48:28.081761 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:48:28.081725    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:48:33.168544 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.168511    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-bpktl"]
Oct 30 18:48:33.168936 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.168700    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/kserve-controller-manager-84d86bb774-bpktl" podUID="6ba78344-8e28-4aca-8240-878d9749c7e5" containerName="manager" containerID="cri-o://b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e" gracePeriod=10
Oct 30 18:48:33.262123 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.262091    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-6hc98"]
Oct 30 18:48:33.265489 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.265468    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.328785 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.328745    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-6hc98"]
Oct 30 18:48:33.378838 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.378809    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pc4r\" (UniqueName: \"kubernetes.io/projected/1e0873a8-a45c-43b0-a1ed-bce0a796a7c5-kube-api-access-2pc4r\") pod \"kserve-controller-manager-84d86bb774-6hc98\" (UID: \"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5\") " pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.378976 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.378846    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e0873a8-a45c-43b0-a1ed-bce0a796a7c5-cert\") pod \"kserve-controller-manager-84d86bb774-6hc98\" (UID: \"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5\") " pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.430251 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.430201    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:48:33.479819 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.479790    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pc4r\" (UniqueName: \"kubernetes.io/projected/1e0873a8-a45c-43b0-a1ed-bce0a796a7c5-kube-api-access-2pc4r\") pod \"kserve-controller-manager-84d86bb774-6hc98\" (UID: \"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5\") " pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.479932 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.479827    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e0873a8-a45c-43b0-a1ed-bce0a796a7c5-cert\") pod \"kserve-controller-manager-84d86bb774-6hc98\" (UID: \"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5\") " pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.482011 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.481993    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e0873a8-a45c-43b0-a1ed-bce0a796a7c5-cert\") pod \"kserve-controller-manager-84d86bb774-6hc98\" (UID: \"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5\") " pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.541293 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.541257    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pc4r\" (UniqueName: \"kubernetes.io/projected/1e0873a8-a45c-43b0-a1ed-bce0a796a7c5-kube-api-access-2pc4r\") pod \"kserve-controller-manager-84d86bb774-6hc98\" (UID: \"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5\") " pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.579927 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.579901    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:33.580268 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.580242    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ba78344-8e28-4aca-8240-878d9749c7e5-cert\" (UniqueName: \"kubernetes.io/secret/6ba78344-8e28-4aca-8240-878d9749c7e5-cert\") pod \"6ba78344-8e28-4aca-8240-878d9749c7e5\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") "
Oct 30 18:48:33.580310 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.580296    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba78344-8e28-4aca-8240-878d9749c7e5-kube-api-access-dkqkk\" (UniqueName: \"kubernetes.io/projected/6ba78344-8e28-4aca-8240-878d9749c7e5-kube-api-access-dkqkk\") pod \"6ba78344-8e28-4aca-8240-878d9749c7e5\" (UID: \"6ba78344-8e28-4aca-8240-878d9749c7e5\") "
Oct 30 18:48:33.582442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.582411    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ba78344-8e28-4aca-8240-878d9749c7e5-cert" pod "6ba78344-8e28-4aca-8240-878d9749c7e5" (UID: "6ba78344-8e28-4aca-8240-878d9749c7e5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:48:33.582605 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.582583    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba78344-8e28-4aca-8240-878d9749c7e5-kube-api-access-dkqkk" pod "6ba78344-8e28-4aca-8240-878d9749c7e5" (UID: "6ba78344-8e28-4aca-8240-878d9749c7e5"). InnerVolumeSpecName "kube-api-access-dkqkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:48:33.681757 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.681690    5025 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ba78344-8e28-4aca-8240-878d9749c7e5-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:33.681757 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.681717    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkqkk\" (UniqueName: \"kubernetes.io/projected/6ba78344-8e28-4aca-8240-878d9749c7e5-kube-api-access-dkqkk\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:48:33.953738 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:33.953648    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-6hc98"]
Oct 30 18:48:33.959737 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:48:33.959714    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0873a8_a45c_43b0_a1ed_bce0a796a7c5.slice/crio-5160ff6d60924a9fc5d41deed9c41a0b51ecf4d2b650217f3f17920d9cfa2cd5 WatchSource:0}: Error finding container 5160ff6d60924a9fc5d41deed9c41a0b51ecf4d2b650217f3f17920d9cfa2cd5: Status 404 returned error can't find the container with id 5160ff6d60924a9fc5d41deed9c41a0b51ecf4d2b650217f3f17920d9cfa2cd5
Oct 30 18:48:34.121276 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.121241    5025 generic.go:356] "Generic (PLEG): container finished" podID="6ba78344-8e28-4aca-8240-878d9749c7e5" containerID="b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e" exitCode=0
Oct 30 18:48:34.121461 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.121285    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84d86bb774-bpktl"
Oct 30 18:48:34.121461 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.121340    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84d86bb774-bpktl" event={"ID":"6ba78344-8e28-4aca-8240-878d9749c7e5","Type":"ContainerDied","Data":"b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e"}
Oct 30 18:48:34.121461 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.121399    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84d86bb774-bpktl" event={"ID":"6ba78344-8e28-4aca-8240-878d9749c7e5","Type":"ContainerDied","Data":"6863b34d2664cd44f61426d1fb90e13d3e7afcd07bdb1c1e7569d92f93ff5cd0"}
Oct 30 18:48:34.121461 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.121430    5025 scope.go:122] "RemoveContainer" containerID="b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e"
Oct 30 18:48:34.122586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.122562    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84d86bb774-6hc98" event={"ID":"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5","Type":"ContainerStarted","Data":"5160ff6d60924a9fc5d41deed9c41a0b51ecf4d2b650217f3f17920d9cfa2cd5"}
Oct 30 18:48:34.129571 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.129554    5025 scope.go:122] "RemoveContainer" containerID="b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e"
Oct 30 18:48:34.129832 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:48:34.129810    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e\": container with ID starting with b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e not found: ID does not exist" containerID="b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e"
Oct 30 18:48:34.129898 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.129839    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e"} err="failed to get container status \"b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e\": rpc error: code = NotFound desc = could not find container \"b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e\": container with ID starting with b84abc90f25ee47d051da7cedf2365893ab16aad518275bcc4277ef00398d02e not found: ID does not exist"
Oct 30 18:48:34.287025 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.286941    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-bpktl"]
Oct 30 18:48:34.334166 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:34.334132    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-84d86bb774-bpktl"]
Oct 30 18:48:35.129165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:35.129130    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84d86bb774-6hc98" event={"ID":"1e0873a8-a45c-43b0-a1ed-bce0a796a7c5","Type":"ContainerStarted","Data":"f46f2d623c0632cbaebb52f0d6c136769cedc9c6aaa5d50b51d0abb6ff1064e6"}
Oct 30 18:48:35.129375 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:35.129216    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:48:35.239585 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:35.239539    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-84d86bb774-6hc98" podStartSLOduration=1.3847780219999999 podStartE2EDuration="2.239525975s" totalImagesPullingTime="854.747953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:35.238936257 +0000 UTC m=+697.319522465" watchObservedRunningTime="2026-10-30 18:48:35.239525975 +0000 UTC m=+697.320112209"
Oct 30 18:48:36.055672 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:48:36.055630    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba78344-8e28-4aca-8240-878d9749c7e5" path="/var/lib/kubelet/pods/6ba78344-8e28-4aca-8240-878d9749c7e5/volumes"
Oct 30 18:48:58.081735 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:48:58.081698    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:49:06.140073 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:06.140043    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-84d86bb774-6hc98"
Oct 30 18:49:08.479234 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.479202    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-56ddc4c448-xqtl2"]
Oct 30 18:49:08.479622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.479569    5025 cpu_manager.go:414] "containerMap: removing container" podUID="6ba78344-8e28-4aca-8240-878d9749c7e5" containerName="manager"
Oct 30 18:49:08.479622 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.479581    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba78344-8e28-4aca-8240-878d9749c7e5" containerName="manager"
Oct 30 18:49:08.479700 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.479656    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba78344-8e28-4aca-8240-878d9749c7e5" containerName="manager"
Oct 30 18:49:08.483839 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.483820    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:08.572508 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.572471    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adc3006b-425b-49c3-8bac-abf0529cfd75-tls-certs\") pod \"model-serving-api-56ddc4c448-xqtl2\" (UID: \"adc3006b-425b-49c3-8bac-abf0529cfd75\") " pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:08.572508 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.572511    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nssz\" (UniqueName: \"kubernetes.io/projected/adc3006b-425b-49c3-8bac-abf0529cfd75-kube-api-access-9nssz\") pod \"model-serving-api-56ddc4c448-xqtl2\" (UID: \"adc3006b-425b-49c3-8bac-abf0529cfd75\") " pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:08.581344 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.581298    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4l24w\""
Oct 30 18:49:08.581344 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.581313    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Oct 30 18:49:08.582958 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.582938    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-xqtl2"]
Oct 30 18:49:08.673669 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.673642    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adc3006b-425b-49c3-8bac-abf0529cfd75-tls-certs\") pod \"model-serving-api-56ddc4c448-xqtl2\" (UID: \"adc3006b-425b-49c3-8bac-abf0529cfd75\") " pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:08.673816 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.673691    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nssz\" (UniqueName: \"kubernetes.io/projected/adc3006b-425b-49c3-8bac-abf0529cfd75-kube-api-access-9nssz\") pod \"model-serving-api-56ddc4c448-xqtl2\" (UID: \"adc3006b-425b-49c3-8bac-abf0529cfd75\") " pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:08.676125 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.676095    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/adc3006b-425b-49c3-8bac-abf0529cfd75-tls-certs\") pod \"model-serving-api-56ddc4c448-xqtl2\" (UID: \"adc3006b-425b-49c3-8bac-abf0529cfd75\") " pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:08.734599 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.734536    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nssz\" (UniqueName: \"kubernetes.io/projected/adc3006b-425b-49c3-8bac-abf0529cfd75-kube-api-access-9nssz\") pod \"model-serving-api-56ddc4c448-xqtl2\" (UID: \"adc3006b-425b-49c3-8bac-abf0529cfd75\") " pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:08.796641 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:08.796618    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:09.170850 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:09.170815    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-56ddc4c448-xqtl2"]
Oct 30 18:49:09.177236 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:49:09.177208    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc3006b_425b_49c3_8bac_abf0529cfd75.slice/crio-4de4cd15d7cd785b6bffb41f8e0c75cb91b26811199403719350ae6eb7ab3af5 WatchSource:0}: Error finding container 4de4cd15d7cd785b6bffb41f8e0c75cb91b26811199403719350ae6eb7ab3af5: Status 404 returned error can't find the container with id 4de4cd15d7cd785b6bffb41f8e0c75cb91b26811199403719350ae6eb7ab3af5
Oct 30 18:49:09.274907 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:09.274871    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-xqtl2" event={"ID":"adc3006b-425b-49c3-8bac-abf0529cfd75","Type":"ContainerStarted","Data":"4de4cd15d7cd785b6bffb41f8e0c75cb91b26811199403719350ae6eb7ab3af5"}
Oct 30 18:49:11.285665 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:11.285575    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-56ddc4c448-xqtl2" event={"ID":"adc3006b-425b-49c3-8bac-abf0529cfd75","Type":"ContainerStarted","Data":"80217d2d9d6afd73b1936e8396904ef6859993ac0d29f156d2ddbbb48c767cc0"}
Oct 30 18:49:11.286028 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:11.285696    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:11.404599 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:11.404546    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/model-serving-api-56ddc4c448-xqtl2" podStartSLOduration=1.605738985 podStartE2EDuration="3.404531703s" totalImagesPullingTime="1.798792718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:11.403106776 +0000 UTC m=+733.483692983" watchObservedRunningTime="2026-10-30 18:49:11.404531703 +0000 UTC m=+733.485117909"
Oct 30 18:49:22.294241 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:22.294208    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-56ddc4c448-xqtl2"
Oct 30 18:49:25.992182 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:25.992145    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-zv8m7"]
Oct 30 18:49:25.998402 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:25.998383    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.050802 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.050780    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Oct 30 18:49:26.098725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.098694    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zv8m7"]
Oct 30 18:49:26.129168 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.129133    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-model-cache\") pod \"s3-init-zv8m7\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") " pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.129350 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.129188    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgk4k\" (UniqueName: \"kubernetes.io/projected/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-kube-api-access-sgk4k\") pod \"s3-init-zv8m7\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") " pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.230440 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.230399    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgk4k\" (UniqueName: \"kubernetes.io/projected/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-kube-api-access-sgk4k\") pod \"s3-init-zv8m7\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") " pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.230630 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.230512    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-model-cache\") pod \"s3-init-zv8m7\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") " pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.230922 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.230902    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-model-cache\") pod \"s3-init-zv8m7\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") " pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.286376 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.286302    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgk4k\" (UniqueName: \"kubernetes.io/projected/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-kube-api-access-sgk4k\") pod \"s3-init-zv8m7\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") " pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.314158 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.314120    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zv8m7"
Oct 30 18:49:26.683960 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:26.683918    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-zv8m7"]
Oct 30 18:49:26.690795 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:49:26.690768    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc71b2d45_b6cb_4cca_b086_fc166aaf4e05.slice/crio-e956cac550839beaf2dcd8e7f13262f6f16648fae6bad570284df05f241d9c39 WatchSource:0}: Error finding container e956cac550839beaf2dcd8e7f13262f6f16648fae6bad570284df05f241d9c39: Status 404 returned error can't find the container with id e956cac550839beaf2dcd8e7f13262f6f16648fae6bad570284df05f241d9c39
Oct 30 18:49:27.355295 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:27.355260    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zv8m7" event={"ID":"c71b2d45-b6cb-4cca-b086-fc166aaf4e05","Type":"ContainerStarted","Data":"e956cac550839beaf2dcd8e7f13262f6f16648fae6bad570284df05f241d9c39"}
Oct 30 18:49:28.084646 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:49:28.084612    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:49:32.378595 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:32.378557    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zv8m7" event={"ID":"c71b2d45-b6cb-4cca-b086-fc166aaf4e05","Type":"ContainerStarted","Data":"43a45de6d3ec94d36ec7d9b54f952ab9823f53cd8020a2d7819afa2bb3ed1c6f"}
Oct 30 18:49:33.384821 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:33.384783    5025 generic.go:356] "Generic (PLEG): container finished" podID="c71b2d45-b6cb-4cca-b086-fc166aaf4e05" containerID="43a45de6d3ec94d36ec7d9b54f952ab9823f53cd8020a2d7819afa2bb3ed1c6f" exitCode=0
Oct 30 18:49:33.385240 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:33.384830    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zv8m7" event={"ID":"c71b2d45-b6cb-4cca-b086-fc166aaf4e05","Type":"ContainerDied","Data":"43a45de6d3ec94d36ec7d9b54f952ab9823f53cd8020a2d7819afa2bb3ed1c6f"}
Oct 30 18:49:38.410469 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:38.410440    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zv8m7" event={"ID":"c71b2d45-b6cb-4cca-b086-fc166aaf4e05","Type":"ContainerStarted","Data":"2e93040ac730aab20b183498aa5a4072feaedeb1bfbacf275eeef4532e329aa9"}
Oct 30 18:49:38.522690 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:49:38.522628    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-zv8m7" podStartSLOduration=2.227597712 podStartE2EDuration="13.522614841s" totalImagesPullingTime="9.685299209s" totalInitContainerRuntime="1.60971792s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:38.521343571 +0000 UTC m=+760.601929772" watchObservedRunningTime="2026-10-30 18:49:38.522614841 +0000 UTC m=+760.603201047"
Oct 30 18:49:58.081419 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:49:58.081385    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:50:28.082596 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:50:28.082561    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:50:43.697267 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:43.697232    5025 generic.go:356] "Generic (PLEG): container finished" podID="c71b2d45-b6cb-4cca-b086-fc166aaf4e05" containerID="2e93040ac730aab20b183498aa5a4072feaedeb1bfbacf275eeef4532e329aa9" exitCode=0
Oct 30 18:50:43.697650 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:43.697301    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zv8m7" event={"ID":"c71b2d45-b6cb-4cca-b086-fc166aaf4e05","Type":"ContainerDied","Data":"2e93040ac730aab20b183498aa5a4072feaedeb1bfbacf275eeef4532e329aa9"}
Oct 30 18:50:44.832783 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:44.832760    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zv8m7"
Oct 30 18:50:44.912708 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:44.912678    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-kube-api-access-sgk4k\" (UniqueName: \"kubernetes.io/projected/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-kube-api-access-sgk4k\") pod \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") "
Oct 30 18:50:44.912829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:44.912747    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-model-cache\") pod \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\" (UID: \"c71b2d45-b6cb-4cca-b086-fc166aaf4e05\") "
Oct 30 18:50:44.914764 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:44.914735    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-kube-api-access-sgk4k" pod "c71b2d45-b6cb-4cca-b086-fc166aaf4e05" (UID: "c71b2d45-b6cb-4cca-b086-fc166aaf4e05"). InnerVolumeSpecName "kube-api-access-sgk4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:50:44.931061 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:44.931025    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-model-cache" pod "c71b2d45-b6cb-4cca-b086-fc166aaf4e05" (UID: "c71b2d45-b6cb-4cca-b086-fc166aaf4e05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:50:45.013351 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:45.013263    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:45.013351 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:45.013289    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgk4k\" (UniqueName: \"kubernetes.io/projected/c71b2d45-b6cb-4cca-b086-fc166aaf4e05-kube-api-access-sgk4k\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:50:45.707191 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:45.707157    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-zv8m7"
Oct 30 18:50:45.707191 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:45.707184    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-zv8m7" event={"ID":"c71b2d45-b6cb-4cca-b086-fc166aaf4e05","Type":"ContainerDied","Data":"e956cac550839beaf2dcd8e7f13262f6f16648fae6bad570284df05f241d9c39"}
Oct 30 18:50:45.707463 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:50:45.707219    5025 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e956cac550839beaf2dcd8e7f13262f6f16648fae6bad570284df05f241d9c39"
Oct 30 18:50:58.081771 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:50:58.081726    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:51:26.377125 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.377087    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"]
Oct 30 18:51:26.377606 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.377483    5025 cpu_manager.go:414] "containerMap: removing container" podUID="c71b2d45-b6cb-4cca-b086-fc166aaf4e05" containerName="download-hf-model"
Oct 30 18:51:26.377606 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.377494    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71b2d45-b6cb-4cca-b086-fc166aaf4e05" containerName="download-hf-model"
Oct 30 18:51:26.377606 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.377509    5025 cpu_manager.go:414] "containerMap: removing container" podUID="c71b2d45-b6cb-4cca-b086-fc166aaf4e05" containerName="s3-init"
Oct 30 18:51:26.377606 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.377514    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c71b2d45-b6cb-4cca-b086-fc166aaf4e05" containerName="s3-init"
Oct 30 18:51:26.377606 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.377572    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c71b2d45-b6cb-4cca-b086-fc166aaf4e05" containerName="s3-init"
Oct 30 18:51:26.380628 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.380610    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.432291 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.432264    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"seaweedfs-s3-creds\""
Oct 30 18:51:26.432446 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.432418    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"cpu-accel-preset-kserve-self-signed-certs\""
Oct 30 18:51:26.432502 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.432492    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"default-dockercfg-cf6ws\""
Oct 30 18:51:26.460921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.460894    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-dshm\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.460921 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.460925    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.461115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.460943    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kserve-provision-location\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.461115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.461026    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-home\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.461115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.461044    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-model-cache\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.461115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.461060    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tmp-dir\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.461115 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.461088    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76jmr\" (UniqueName: \"kubernetes.io/projected/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kube-api-access-76jmr\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.479603 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.479573    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"openshift-service-ca.crt\""
Oct 30 18:51:26.479757 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.479702    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7\"/\"kube-root-ca.crt\""
Oct 30 18:51:26.481270 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.481251    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"]
Oct 30 18:51:26.561736 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.561706    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76jmr\" (UniqueName: \"kubernetes.io/projected/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kube-api-access-76jmr\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.561913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.561778    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-dshm\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.561913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.561810    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.561913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.561834    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kserve-provision-location\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.561913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.561873    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-home\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.561913 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.561894    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-model-cache\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.562141 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.561916    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tmp-dir\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.562374 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.562314    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kserve-provision-location\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.562374 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.562360    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tmp-dir\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.562559 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.562437    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-model-cache\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.562559 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.562529    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-home\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.564364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.564346    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-dshm\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.564550 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.564532    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.618564 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.618529    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76jmr\" (UniqueName: \"kubernetes.io/projected/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kube-api-access-76jmr\") pod \"cpu-accel-preset-kserve-6d76ccc7fc-zd6v6\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") " pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:26.695696 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:26.695623    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:27.122722 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:27.122697    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"]
Oct 30 18:51:27.128854 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:51:27.128829    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27d6ed7_af66_4351_8e6e_5116bc0a942d.slice/crio-b713747155ace9bde78f3d3974ab940e3aa386ef60a8f5c1abca73e26835472c WatchSource:0}: Error finding container b713747155ace9bde78f3d3974ab940e3aa386ef60a8f5c1abca73e26835472c: Status 404 returned error can't find the container with id b713747155ace9bde78f3d3974ab940e3aa386ef60a8f5c1abca73e26835472c
Oct 30 18:51:27.891033 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:27.890998    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" event={"ID":"c27d6ed7-af66-4351-8e6e-5116bc0a942d","Type":"ContainerStarted","Data":"abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001"}
Oct 30 18:51:27.891438 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:27.891043    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" event={"ID":"c27d6ed7-af66-4351-8e6e-5116bc0a942d","Type":"ContainerStarted","Data":"b713747155ace9bde78f3d3974ab940e3aa386ef60a8f5c1abca73e26835472c"}
Oct 30 18:51:28.038562 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:28.038532    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"]
Oct 30 18:51:28.083675 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:28.083573    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:51:28.982607 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:28.982570    5025 secret.go:189] Couldn't get secret e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-self-signed-certs: secret "cpu-accel-preset-kserve-self-signed-certs" not found
Oct 30 18:51:28.983453 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:28.982690    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs podName:c27d6ed7-af66-4351-8e6e-5116bc0a942d nodeName:}" failed. No retries permitted until 2026-10-30 18:51:29.482632077 +0000 UTC m=+871.563218286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs") pod "cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d") : secret "cpu-accel-preset-kserve-self-signed-certs" not found
Oct 30 18:51:29.487802 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:29.487765    5025 secret.go:189] Couldn't get secret e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-self-signed-certs: secret "cpu-accel-preset-kserve-self-signed-certs" not found
Oct 30 18:51:29.488031 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:29.487840    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs podName:c27d6ed7-af66-4351-8e6e-5116bc0a942d nodeName:}" failed. No retries permitted until 2026-10-30 18:51:30.487827552 +0000 UTC m=+872.568413741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs") pod "cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d") : secret "cpu-accel-preset-kserve-self-signed-certs" not found
Oct 30 18:51:29.899998 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:29.899943    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" podUID="c27d6ed7-af66-4351-8e6e-5116bc0a942d" containerName="storage-initializer" containerID="cri-o://abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001" gracePeriod=60
Oct 30 18:51:30.496265 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:30.496222    5025 secret.go:189] Couldn't get secret e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-self-signed-certs: secret "cpu-accel-preset-kserve-self-signed-certs" not found
Oct 30 18:51:30.496781 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:30.496312    5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs podName:c27d6ed7-af66-4351-8e6e-5116bc0a942d nodeName:}" failed. No retries permitted until 2026-10-30 18:51:32.496291452 +0000 UTC m=+874.576877661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs") pod "cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d") : secret "cpu-accel-preset-kserve-self-signed-certs" not found
Oct 30 18:51:32.065646 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.065589    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:32.111829 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.111801    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tmp-dir\") pod \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") "
Oct 30 18:51:32.111977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.111845    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kserve-provision-location\") pod \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") "
Oct 30 18:51:32.111977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.111890    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs\" (UniqueName: \"kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs\") pod \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") "
Oct 30 18:51:32.111977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.111935    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kube-api-access-76jmr\" (UniqueName: \"kubernetes.io/projected/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kube-api-access-76jmr\") pod \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") "
Oct 30 18:51:32.111977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.111956    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-model-cache\") pod \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") "
Oct 30 18:51:32.111977 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.111979    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-dshm\") pod \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") "
Oct 30 18:51:32.112215 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.111995    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-home\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-home\") pod \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\" (UID: \"c27d6ed7-af66-4351-8e6e-5116bc0a942d\") "
Oct 30 18:51:32.112215 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.112094    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tmp-dir" pod "c27d6ed7-af66-4351-8e6e-5116bc0a942d" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:32.112341 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.112235    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.112341 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.112300    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-model-cache" pod "c27d6ed7-af66-4351-8e6e-5116bc0a942d" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:32.112529 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.112504    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-home" pod "c27d6ed7-af66-4351-8e6e-5116bc0a942d" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:32.114129 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.114108    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs" pod "c27d6ed7-af66-4351-8e6e-5116bc0a942d" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:51:32.114129 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.114119    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-dshm" pod "c27d6ed7-af66-4351-8e6e-5116bc0a942d" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:32.114551 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.114534    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kube-api-access-76jmr" pod "c27d6ed7-af66-4351-8e6e-5116bc0a942d" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d"). InnerVolumeSpecName "kube-api-access-76jmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:51:32.175155 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.175111    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kserve-provision-location" pod "c27d6ed7-af66-4351-8e6e-5116bc0a942d" (UID: "c27d6ed7-af66-4351-8e6e-5116bc0a942d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:51:32.213331 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.213297    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.213443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.213341    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c27d6ed7-af66-4351-8e6e-5116bc0a942d-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.213443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.213357    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76jmr\" (UniqueName: \"kubernetes.io/projected/c27d6ed7-af66-4351-8e6e-5116bc0a942d-kube-api-access-76jmr\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.213443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.213374    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.213443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.213386    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.213443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.213394    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c27d6ed7-af66-4351-8e6e-5116bc0a942d-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:51:32.914533 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.914497    5025 generic.go:356] "Generic (PLEG): container finished" podID="c27d6ed7-af66-4351-8e6e-5116bc0a942d" containerID="abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001" exitCode=0
Oct 30 18:51:32.914686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.914535    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" event={"ID":"c27d6ed7-af66-4351-8e6e-5116bc0a942d","Type":"ContainerDied","Data":"abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001"}
Oct 30 18:51:32.914686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.914549    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"
Oct 30 18:51:32.914686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.914577    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6" event={"ID":"c27d6ed7-af66-4351-8e6e-5116bc0a942d","Type":"ContainerDied","Data":"b713747155ace9bde78f3d3974ab940e3aa386ef60a8f5c1abca73e26835472c"}
Oct 30 18:51:32.914686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.914600    5025 scope.go:122] "RemoveContainer" containerID="abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001"
Oct 30 18:51:32.924592 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.924454    5025 scope.go:122] "RemoveContainer" containerID="abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001"
Oct 30 18:51:32.924724 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:32.924705    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001\": container with ID starting with abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001 not found: ID does not exist" containerID="abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001"
Oct 30 18:51:32.924764 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:32.924733    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001"} err="failed to get container status \"abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001\": rpc error: code = NotFound desc = could not find container \"abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001\": container with ID starting with abdeb53be796761e6aab42e0c1c46bc80ae66face65ff7d065b824423cca6001 not found: ID does not exist"
Oct 30 18:51:33.190731 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:33.190649    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"]
Oct 30 18:51:33.197892 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:33.197867    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-applies-to-workload-8e506af7/cpu-accel-preset-kserve-6d76ccc7fc-zd6v6"]
Oct 30 18:51:34.056196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:34.056163    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c27d6ed7-af66-4351-8e6e-5116bc0a942d" path="/var/lib/kubelet/pods/c27d6ed7-af66-4351-8e6e-5116bc0a942d/volumes"
Oct 30 18:51:43.180136 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.180094    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"]
Oct 30 18:51:43.180667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.180503    5025 cpu_manager.go:414] "containerMap: removing container" podUID="c27d6ed7-af66-4351-8e6e-5116bc0a942d" containerName="storage-initializer"
Oct 30 18:51:43.180667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.180516    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c27d6ed7-af66-4351-8e6e-5116bc0a942d" containerName="storage-initializer"
Oct 30 18:51:43.180667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.180593    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c27d6ed7-af66-4351-8e6e-5116bc0a942d" containerName="storage-initializer"
Oct 30 18:51:43.183766 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.183746    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.234108 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.234080    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"cpu-accel-precedence-kserve-self-signed-certs\""
Oct 30 18:51:43.281462 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.281432    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"kube-root-ca.crt\""
Oct 30 18:51:43.281462 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.281433    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"default-dockercfg-xjsk4\""
Oct 30 18:51:43.281667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.281477    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"openshift-service-ca.crt\""
Oct 30 18:51:43.281667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.281498    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-user-image-wins-06960b42\"/\"seaweedfs-s3-creds\""
Oct 30 18:51:43.283056 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.283037    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"]
Oct 30 18:51:43.308946 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.308922    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-tmp-dir\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.309063 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.308959    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.309099 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.309052    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5be3c05-646f-4a31-bdd0-cb752216517a-tls-certs\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.309147 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.309098    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-dshm\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.309147 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.309119    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-home\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.309287 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.309266    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-model-cache\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.309374 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.309304    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j78pn\" (UniqueName: \"kubernetes.io/projected/a5be3c05-646f-4a31-bdd0-cb752216517a-kube-api-access-j78pn\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410051 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410023    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5be3c05-646f-4a31-bdd0-cb752216517a-tls-certs\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410234 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410062    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-dshm\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410234 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410080    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-home\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410234 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410192    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-model-cache\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410234 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410224    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j78pn\" (UniqueName: \"kubernetes.io/projected/a5be3c05-646f-4a31-bdd0-cb752216517a-kube-api-access-j78pn\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410466 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410271    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-tmp-dir\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410466 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410364    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410573 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410483    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-home\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.410655 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.410631    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-model-cache\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.411044 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.411018    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-kserve-provision-location\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.411234 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.411213    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-tmp-dir\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.413027 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.412999    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-dshm\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.413189 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.413170    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5be3c05-646f-4a31-bdd0-cb752216517a-tls-certs\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.466222 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.466195    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j78pn\" (UniqueName: \"kubernetes.io/projected/a5be3c05-646f-4a31-bdd0-cb752216517a-kube-api-access-j78pn\") pod \"cpu-accel-precedence-kserve-7b79794789-tc8sr\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") " pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:43.500331 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:43.500300    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:51:44.034171 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:51:44.034138    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5be3c05_646f_4a31_bdd0_cb752216517a.slice/crio-8522b7542e9fd24c671477e2168d80c45bc75313596e01e9a38e3de759aed6a1 WatchSource:0}: Error finding container 8522b7542e9fd24c671477e2168d80c45bc75313596e01e9a38e3de759aed6a1: Status 404 returned error can't find the container with id 8522b7542e9fd24c671477e2168d80c45bc75313596e01e9a38e3de759aed6a1
Oct 30 18:51:44.035267 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:44.035244    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"]
Oct 30 18:51:44.969000 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:44.968969    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" event={"ID":"a5be3c05-646f-4a31-bdd0-cb752216517a","Type":"ContainerStarted","Data":"fb9c5987218883044a17ca763acdde0eafd06beb0bdcc574bca0632fe88e448d"}
Oct 30 18:51:44.969416 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:44.969008    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" event={"ID":"a5be3c05-646f-4a31-bdd0-cb752216517a","Type":"ContainerStarted","Data":"8522b7542e9fd24c671477e2168d80c45bc75313596e01e9a38e3de759aed6a1"}
Oct 30 18:51:47.936657 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:47.936621    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"]
Oct 30 18:51:47.941314 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:47.941292    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:47.992525 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:47.992501    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-dstgc\""
Oct 30 18:51:48.039373 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.039344    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 18:51:48.041162 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.041131    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"]
Oct 30 18:51:48.054157 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054128    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054157 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054164    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054394 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054236    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054394 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054267    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmqmm\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-kube-api-access-tmqmm\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054386    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054426    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054517    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054539    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.054574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.054561    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155636 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155605    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155706    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155731    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155747    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmqmm\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-kube-api-access-tmqmm\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155801 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155774    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155996 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155800    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155996 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155913    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.155996 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.155949    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.156159 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.156092    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.156214 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.156191    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.156214 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.156203    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.156412 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.156388    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.156518 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.156463    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.156614 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.156594    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.157918 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.157897    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.158471 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.158455    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.212964 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.212894    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.212964 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.212917    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmqmm\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-kube-api-access-tmqmm\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-9xt5b\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.254422 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.254398    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:48.718593 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.718559    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"]
Oct 30 18:51:48.735208 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:51:48.735180    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486a72db_72e2_40eb_9d6a_77ca411b7fd7.slice/crio-34be035f277e93f845025568d1cfc46c2389d92abdee27c13777a7ad68d85701 WatchSource:0}: Error finding container 34be035f277e93f845025568d1cfc46c2389d92abdee27c13777a7ad68d85701: Status 404 returned error can't find the container with id 34be035f277e93f845025568d1cfc46c2389d92abdee27c13777a7ad68d85701
Oct 30 18:51:48.737250 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.737235    5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:51:48.994101 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:48.994022    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" event={"ID":"486a72db-72e2-40eb-9d6a-77ca411b7fd7","Type":"ContainerStarted","Data":"34be035f277e93f845025568d1cfc46c2389d92abdee27c13777a7ad68d85701"}
Oct 30 18:51:50.000283 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:50.000249    5025 generic.go:356] "Generic (PLEG): container finished" podID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerID="fb9c5987218883044a17ca763acdde0eafd06beb0bdcc574bca0632fe88e448d" exitCode=0
Oct 30 18:51:50.000685 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:50.000342    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" event={"ID":"a5be3c05-646f-4a31-bdd0-cb752216517a","Type":"ContainerDied","Data":"fb9c5987218883044a17ca763acdde0eafd06beb0bdcc574bca0632fe88e448d"}
Oct 30 18:51:54.026464 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:54.026424    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:51:54.026974 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:54.026527    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:51:54.026974 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:54.026559    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:51:55.042119 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:55.042051    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" event={"ID":"486a72db-72e2-40eb-9d6a-77ca411b7fd7","Type":"ContainerStarted","Data":"4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7"}
Oct 30 18:51:55.254992 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:55.254924    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:51:55.256395 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:55.256314    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:55.256515 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:55.256426    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:56.255342 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:56.255244    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:56.255342 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:56.255294    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:57.255215 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:57.255176    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:57.255422 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:57.255232    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:57.832030 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:57.831980    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podStartSLOduration=5.543940976 podStartE2EDuration="10.831964679s" totalImagesPullingTime="5.288023703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:55.21688058 +0000 UTC m=+897.297466806" watchObservedRunningTime="2026-10-30 18:51:57.831964679 +0000 UTC m=+899.912550887"
Oct 30 18:51:57.833021 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:57.832996    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"]
Oct 30 18:51:58.112855 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:51:58.112783    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:51:58.204377 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:58.204351    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:51:58.210838 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:58.210667    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:51:58.255524 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:58.255077    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:58.255524 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:58.255139    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:51:59.255796 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:59.255762    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 18:51:59.256300 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:51:59.255832    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 18:52:00.329553 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:00.329519    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:52:00.330663 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:00.330638    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:52:00.331302 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:00.331282    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 18:52:21.207100 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:21.207063    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" event={"ID":"a5be3c05-646f-4a31-bdd0-cb752216517a","Type":"ContainerStarted","Data":"d4f27ed35aab6f7f1008ffda25407f5ffcf7b169cac73ca4164b680bf62506da"}
Oct 30 18:52:21.418392 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:21.418341    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" podStartSLOduration=2.075202664 podStartE2EDuration="38.41830565s" totalImagesPullingTime="31.052140766s" totalInitContainerRuntime="5.29096222s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:51:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:21.41757962 +0000 UTC m=+923.498165845" watchObservedRunningTime="2026-10-30 18:52:21.41830565 +0000 UTC m=+923.498891856"
Oct 30 18:52:21.501131 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:21.501096    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:52:28.086227 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:52:28.086191    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:52:32.900577 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:32.900542    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"]
Oct 30 18:52:33.119733 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.119701    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"]
Oct 30 18:52:33.119903 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.119802    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.198878 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.198794    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.198878 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.198833    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.198878 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.198850    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.198878 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.198876    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.199134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.198952    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gljh\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-kube-api-access-2gljh\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.199134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.199117    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.199246 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.199148    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.199246 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.199178    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.199338 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.199279    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.300634 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.300596    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.300826 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.300641    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gljh\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-kube-api-access-2gljh\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.300826 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.300731    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.300826 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.300758    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301071 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.300821    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301225 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301095    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301225 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301110    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301225 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301212    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301424 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301251    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301424 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301260    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301424 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301282    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301424 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301400    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301633 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301552    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.301966 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.301945    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.303613 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.303582    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.303772 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.303756    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.359378 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.359340    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gljh\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-kube-api-access-2gljh\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.359551 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.359523    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-r79wl\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.461868 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.461755    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:33.986667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:33.986615    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"]
Oct 30 18:52:34.026192 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:52:34.026160    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401ed0a9_2ba8_403e_82b0_8d73081b8b5e.slice/crio-0f830025c74aecc324aca1b32cfb812b38c2ed5fc4dc6110c2e080d8ec27aab5 WatchSource:0}: Error finding container 0f830025c74aecc324aca1b32cfb812b38c2ed5fc4dc6110c2e080d8ec27aab5: Status 404 returned error can't find the container with id 0f830025c74aecc324aca1b32cfb812b38c2ed5fc4dc6110c2e080d8ec27aab5
Oct 30 18:52:34.028772 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:34.028732    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:34.028886 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:34.028834    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:34.028886 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:34.028880    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:52:34.269297 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:34.269193    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" event={"ID":"401ed0a9-2ba8-403e-82b0-8d73081b8b5e","Type":"ContainerStarted","Data":"47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33"}
Oct 30 18:52:34.269297 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:34.269265    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" event={"ID":"401ed0a9-2ba8-403e-82b0-8d73081b8b5e","Type":"ContainerStarted","Data":"0f830025c74aecc324aca1b32cfb812b38c2ed5fc4dc6110c2e080d8ec27aab5"}
Oct 30 18:52:34.384584 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:34.384513    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" podStartSLOduration=2.384494666 podStartE2EDuration="2.384494666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:34.382938893 +0000 UTC m=+936.463525119" watchObservedRunningTime="2026-10-30 18:52:34.384494666 +0000 UTC m=+936.465080873"
Oct 30 18:52:34.461956 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:34.461913    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:35.468764 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:35.468738    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:36.230306 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:36.230262    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" podUID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerName="main" containerID="cri-o://d4f27ed35aab6f7f1008ffda25407f5ffcf7b169cac73ca4164b680bf62506da" gracePeriod=45
Oct 30 18:52:36.279044 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:36.279014    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:36.279956 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:52:36.279934    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:52:58.082924 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:52:58.082882    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:53:19.500687 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:19.500653    5025 generic.go:356] "Generic (PLEG): container finished" podID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerID="47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33" exitCode=0
Oct 30 18:53:19.501095 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:19.500702    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" event={"ID":"401ed0a9-2ba8-403e-82b0-8d73081b8b5e","Type":"ContainerDied","Data":"47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33"}
Oct 30 18:53:19.501137 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:19.501096    5025 scope.go:122] "RemoveContainer" containerID="47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33"
Oct 30 18:53:19.502263 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:19.502231    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:19.502369 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:19.502343    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:19.502413 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:19.502392    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:20.511717 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:20.511687    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" event={"ID":"401ed0a9-2ba8-403e-82b0-8d73081b8b5e","Type":"ContainerStarted","Data":"22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80"}
Oct 30 18:53:21.282507 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.282415    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:53:21.464578 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.464483    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:53:21.466232 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.466207    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-r79wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 18:53:21.483437 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.483384    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 18:53:21.517471 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.517405    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7b79794789-tc8sr_a5be3c05-646f-4a31-bdd0-cb752216517a/main/0.log"
Oct 30 18:53:21.517824 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.517776    5025 generic.go:356] "Generic (PLEG): container finished" podID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerID="d4f27ed35aab6f7f1008ffda25407f5ffcf7b169cac73ca4164b680bf62506da" exitCode=137
Oct 30 18:53:21.517869 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.517848    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" event={"ID":"a5be3c05-646f-4a31-bdd0-cb752216517a","Type":"ContainerDied","Data":"d4f27ed35aab6f7f1008ffda25407f5ffcf7b169cac73ca4164b680bf62506da"}
Oct 30 18:53:21.912803 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.912781    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7b79794789-tc8sr_a5be3c05-646f-4a31-bdd0-cb752216517a/main/0.log"
Oct 30 18:53:21.915332 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:21.915307    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:53:22.006047 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006012    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5be3c05-646f-4a31-bdd0-cb752216517a-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5be3c05-646f-4a31-bdd0-cb752216517a-tls-certs\") pod \"a5be3c05-646f-4a31-bdd0-cb752216517a\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") "
Oct 30 18:53:22.006207 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006064    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-dshm\") pod \"a5be3c05-646f-4a31-bdd0-cb752216517a\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") "
Oct 30 18:53:22.006207 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006099    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-home\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-home\") pod \"a5be3c05-646f-4a31-bdd0-cb752216517a\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") "
Oct 30 18:53:22.006207 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006152    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-model-cache\") pod \"a5be3c05-646f-4a31-bdd0-cb752216517a\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") "
Oct 30 18:53:22.006443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006219    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-tmp-dir\") pod \"a5be3c05-646f-4a31-bdd0-cb752216517a\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") "
Oct 30 18:53:22.006443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006245    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5be3c05-646f-4a31-bdd0-cb752216517a-kube-api-access-j78pn\" (UniqueName: \"kubernetes.io/projected/a5be3c05-646f-4a31-bdd0-cb752216517a-kube-api-access-j78pn\") pod \"a5be3c05-646f-4a31-bdd0-cb752216517a\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") "
Oct 30 18:53:22.006443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006277    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-kserve-provision-location\") pod \"a5be3c05-646f-4a31-bdd0-cb752216517a\" (UID: \"a5be3c05-646f-4a31-bdd0-cb752216517a\") "
Oct 30 18:53:22.006443 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006425    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-model-cache" pod "a5be3c05-646f-4a31-bdd0-cb752216517a" (UID: "a5be3c05-646f-4a31-bdd0-cb752216517a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:22.006647 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006498    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-home" pod "a5be3c05-646f-4a31-bdd0-cb752216517a" (UID: "a5be3c05-646f-4a31-bdd0-cb752216517a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:22.006647 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006641    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:22.006756 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006660    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:22.006756 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.006708    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-tmp-dir" pod "a5be3c05-646f-4a31-bdd0-cb752216517a" (UID: "a5be3c05-646f-4a31-bdd0-cb752216517a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:22.008412 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.008384    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5be3c05-646f-4a31-bdd0-cb752216517a-tls-certs" pod "a5be3c05-646f-4a31-bdd0-cb752216517a" (UID: "a5be3c05-646f-4a31-bdd0-cb752216517a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:53:22.008761 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.008742    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5be3c05-646f-4a31-bdd0-cb752216517a-kube-api-access-j78pn" pod "a5be3c05-646f-4a31-bdd0-cb752216517a" (UID: "a5be3c05-646f-4a31-bdd0-cb752216517a"). InnerVolumeSpecName "kube-api-access-j78pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:53:22.008833 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.008770    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-dshm" pod "a5be3c05-646f-4a31-bdd0-cb752216517a" (UID: "a5be3c05-646f-4a31-bdd0-cb752216517a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:22.087127 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.087025    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-kserve-provision-location" pod "a5be3c05-646f-4a31-bdd0-cb752216517a" (UID: "a5be3c05-646f-4a31-bdd0-cb752216517a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:53:22.107862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.107830    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:22.107862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.107858    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j78pn\" (UniqueName: \"kubernetes.io/projected/a5be3c05-646f-4a31-bdd0-cb752216517a-kube-api-access-j78pn\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:22.108041 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.107869    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:22.108041 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.107878    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5be3c05-646f-4a31-bdd0-cb752216517a-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:22.108041 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.107887    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5be3c05-646f-4a31-bdd0-cb752216517a-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:53:22.463184 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.462722    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-r79wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 18:53:22.463184 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.462784    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 18:53:22.523537 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.523509    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-cpu-accelerator-preset-user-image-wins-06960b42_cpu-accel-precedence-kserve-7b79794789-tc8sr_a5be3c05-646f-4a31-bdd0-cb752216517a/main/0.log"
Oct 30 18:53:22.523971 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.523952    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"
Oct 30 18:53:22.524041 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.523950    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr" event={"ID":"a5be3c05-646f-4a31-bdd0-cb752216517a","Type":"ContainerDied","Data":"8522b7542e9fd24c671477e2168d80c45bc75313596e01e9a38e3de759aed6a1"}
Oct 30 18:53:22.524041 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.524012    5025 scope.go:122] "RemoveContainer" containerID="d4f27ed35aab6f7f1008ffda25407f5ffcf7b169cac73ca4164b680bf62506da"
Oct 30 18:53:22.532677 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.532484    5025 scope.go:122] "RemoveContainer" containerID="fb9c5987218883044a17ca763acdde0eafd06beb0bdcc574bca0632fe88e448d"
Oct 30 18:53:22.693437 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.693403    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"]
Oct 30 18:53:22.740393 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:22.740308    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-user-image-wins-06960b42/cpu-accel-precedence-kserve-7b79794789-tc8sr"]
Oct 30 18:53:23.474345 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:23.465754    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-r79wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 18:53:23.474345 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:23.465815    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 18:53:24.056137 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:24.056104    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5be3c05-646f-4a31-bdd0-cb752216517a" path="/var/lib/kubelet/pods/a5be3c05-646f-4a31-bdd0-cb752216517a/volumes"
Oct 30 18:53:24.472347 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:24.463709    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-r79wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 18:53:24.472347 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:24.463782    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 18:53:25.462550 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:25.462514    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-r79wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 18:53:25.463017 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:25.462609    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 18:53:26.467022 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:26.466991    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:53:26.467968 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:26.467948    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 18:53:28.082176 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:53:28.082138    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:53:29.993973 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:29.993937    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"]
Oct 30 18:53:29.994734 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:29.994704    5025 cpu_manager.go:414] "containerMap: removing container" podUID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerName="main"
Oct 30 18:53:29.994734 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:29.994735    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerName="main"
Oct 30 18:53:29.994891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:29.994760    5025 cpu_manager.go:414] "containerMap: removing container" podUID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerName="storage-initializer"
Oct 30 18:53:29.994891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:29.994769    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerName="storage-initializer"
Oct 30 18:53:29.994955 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:29.994900    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5be3c05-646f-4a31-bdd0-cb752216517a" containerName="main"
Oct 30 18:53:30.000344 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.000310    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.075937 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.075907    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.076108 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.075952    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.076108 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.076035    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9t4h\" (UniqueName: \"kubernetes.io/projected/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kube-api-access-t9t4h\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.076205 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.076107    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.076205 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.076175    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.076337 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.076237    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.076337 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.076268    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.076441 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.076346    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.100863 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.100836    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"openshift-service-ca.crt\""
Oct 30 18:53:30.101020 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.100836    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"kube-root-ca.crt\""
Oct 30 18:53:30.101020 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.100835    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 18:53:30.101020 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.100835    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"seaweedfs-s3-creds\""
Oct 30 18:53:30.101020 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.100928    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"default-dockercfg-kvpjm\""
Oct 30 18:53:30.101020 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.100854    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cb25a14c9ba423627edeb5388343bd353-kserve-self-signed-certs\""
Oct 30 18:53:30.102640 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.102618    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"]
Oct 30 18:53:30.177764 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.177718    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.177944 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.177826    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.177944 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.177867    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.177944 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.177898    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9t4h\" (UniqueName: \"kubernetes.io/projected/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kube-api-access-t9t4h\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.177959    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178032    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178059    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178093    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178733 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178287    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-home\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178733 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178429    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kserve-provision-location\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178733 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178546    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tmp-dir\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178733 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178595    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-model-cache\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.178733 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.178615    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-cabundle-cert\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.180596 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.180575    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tls-certs\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.180691 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.180624    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-dshm\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.234547 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.234512    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9t4h\" (UniqueName: \"kubernetes.io/projected/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kube-api-access-t9t4h\") pod \"test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.315459 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.315393    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:53:30.420958 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.420928    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"]
Oct 30 18:53:30.424742 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.424721    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.476142 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.476119    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c\"/\"test-cpu-accelerator-preset-ser-afc820a4-epp-sa-dockercfg-fxllp\""
Oct 30 18:53:30.482055 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.482030    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs296\" (UniqueName: \"kubernetes.io/projected/791fd137-c4e7-49cb-954d-c3c447d3b156-kube-api-access-xs296\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.482134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.482071    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791fd137-c4e7-49cb-954d-c3c447d3b156-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.483542 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.483523    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"]
Oct 30 18:53:30.583407 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.583315    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs296\" (UniqueName: \"kubernetes.io/projected/791fd137-c4e7-49cb-954d-c3c447d3b156-kube-api-access-xs296\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.583407 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.583373    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791fd137-c4e7-49cb-954d-c3c447d3b156-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.585846 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.585822    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791fd137-c4e7-49cb-954d-c3c447d3b156-tls-certs\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.639893 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.639862    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs296\" (UniqueName: \"kubernetes.io/projected/791fd137-c4e7-49cb-954d-c3c447d3b156-kube-api-access-xs296\") pod \"test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") " pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.738018 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.737981    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:30.808249 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:30.808221    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"]
Oct 30 18:53:30.816345 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:53:30.816298    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3262dee_f035_4c5e_b9a2_5eaf83dffb84.slice/crio-66ad6da919345e09e9959575febd4ffa61798744caaf43beecbf3240f7cfa894 WatchSource:0}: Error finding container 66ad6da919345e09e9959575febd4ffa61798744caaf43beecbf3240f7cfa894: Status 404 returned error can't find the container with id 66ad6da919345e09e9959575febd4ffa61798744caaf43beecbf3240f7cfa894
Oct 30 18:53:31.106275 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:31.106187    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"]
Oct 30 18:53:31.112090 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:53:31.112059    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791fd137_c4e7_49cb_954d_c3c447d3b156.slice/crio-f2bb4acc3db66c1ba7ae79b7c42d7c35931fe1107c4a371d98ffaaa1fba8eeec WatchSource:0}: Error finding container f2bb4acc3db66c1ba7ae79b7c42d7c35931fe1107c4a371d98ffaaa1fba8eeec: Status 404 returned error can't find the container with id f2bb4acc3db66c1ba7ae79b7c42d7c35931fe1107c4a371d98ffaaa1fba8eeec
Oct 30 18:53:31.568557 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:31.568521    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" event={"ID":"791fd137-c4e7-49cb-954d-c3c447d3b156","Type":"ContainerStarted","Data":"f2bb4acc3db66c1ba7ae79b7c42d7c35931fe1107c4a371d98ffaaa1fba8eeec"}
Oct 30 18:53:31.570073 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:31.570047    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" event={"ID":"d3262dee-f035-4c5e-b9a2-5eaf83dffb84","Type":"ContainerStarted","Data":"57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514"}
Oct 30 18:53:31.570213 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:31.570085    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" event={"ID":"d3262dee-f035-4c5e-b9a2-5eaf83dffb84","Type":"ContainerStarted","Data":"66ad6da919345e09e9959575febd4ffa61798744caaf43beecbf3240f7cfa894"}
Oct 30 18:53:33.172466 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.172430    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"]
Oct 30 18:53:33.176080 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.176061    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213513    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213597    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213659    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213721    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213772    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213808    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tr8r\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-kube-api-access-2tr8r\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213843    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213866    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.216379 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.213900    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315451 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315426    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315473    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315508    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315539    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315557    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tr8r\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-kube-api-access-2tr8r\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315797 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315585    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315797 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315606    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315797 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315635    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.315797 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315661    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.316019 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.315927    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.316387 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.316309    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.316583 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.316554    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.316583 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.316576    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.316747 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.316701    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.317940 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.317912    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.318849 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.318826    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.375032 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.374997    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tr8r\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-kube-api-access-2tr8r\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.375205 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.375047    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-fcdrl\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.435552 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.435480    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"]
Oct 30 18:53:33.493380 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.493348    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:33.583355 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.583295    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" event={"ID":"791fd137-c4e7-49cb-954d-c3c447d3b156","Type":"ContainerStarted","Data":"4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa"}
Oct 30 18:53:33.584052 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.584026    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:53:33.584909 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.584879    5025 generic.go:356] "Generic (PLEG): container finished" podID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerID="57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514" exitCode=0
Oct 30 18:53:33.585042 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.584943    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" event={"ID":"d3262dee-f035-4c5e-b9a2-5eaf83dffb84","Type":"ContainerDied","Data":"57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514"}
Oct 30 18:53:33.694085 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.693967    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" podStartSLOduration=1.603990678 podStartE2EDuration="3.693951851s" totalImagesPullingTime="2.089961173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:33.693026387 +0000 UTC m=+995.773612613" watchObservedRunningTime="2026-10-30 18:53:33.693951851 +0000 UTC m=+995.774538064"
Oct 30 18:53:33.918370 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.918338    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"]
Oct 30 18:53:33.924483 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:53:33.924456    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64168401_3f30_4ea0_939c_5f2ba1cab7f1.slice/crio-9f8e2739b067f2ea3cc734dc07f882593ab3f23f65b02e66bc981f0c43f5019e WatchSource:0}: Error finding container 9f8e2739b067f2ea3cc734dc07f882593ab3f23f65b02e66bc981f0c43f5019e: Status 404 returned error can't find the container with id 9f8e2739b067f2ea3cc734dc07f882593ab3f23f65b02e66bc981f0c43f5019e
Oct 30 18:53:33.926888 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.926857    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:33.926941 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.926925    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:33.926976 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:33.926957    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 18:53:34.590732 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:34.590688    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" event={"ID":"64168401-3f30-4ea0-939c-5f2ba1cab7f1","Type":"ContainerStarted","Data":"c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5"}
Oct 30 18:53:34.591221 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:34.590755    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" event={"ID":"64168401-3f30-4ea0-939c-5f2ba1cab7f1","Type":"ContainerStarted","Data":"9f8e2739b067f2ea3cc734dc07f882593ab3f23f65b02e66bc981f0c43f5019e"}
Oct 30 18:53:34.705295 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:34.705243    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podStartSLOduration=1.705229168 podStartE2EDuration="1.705229168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:34.705053253 +0000 UTC m=+996.785639483" watchObservedRunningTime="2026-10-30 18:53:34.705229168 +0000 UTC m=+996.785815375"
Oct 30 18:53:35.493579 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:35.493539    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:35.495133 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:35.495105    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:35.495246 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:35.495163    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 18:53:36.494567 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:36.494530    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:36.494959 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:36.494600    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 18:53:37.493830 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:37.493791    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:37.494009 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:37.493853    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 18:53:38.494010 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:38.493973    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:38.494502 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:38.494032    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 18:53:39.493782 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:39.493743    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 30 18:53:39.493954 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:39.493803    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 30 18:53:40.498455 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:40.498424    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:40.498826 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:40.498791    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:40.499534 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:53:40.499515    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 18:53:58.092918 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:53:58.092878    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:54:05.597731 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:54:05.597700    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:54:28.086210 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:54:28.086112    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:54:58.088111 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:54:58.088073    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:55:18.116540 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:18.116505    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" event={"ID":"d3262dee-f035-4c5e-b9a2-5eaf83dffb84","Type":"ContainerStarted","Data":"809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5"}
Oct 30 18:55:18.316408 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:18.316304    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:55:18.342009 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:18.341952    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podStartSLOduration=2.66116039 podStartE2EDuration="1m49.34193849s" totalImagesPullingTime="1m44.366990799s" totalInitContainerRuntime="2.313787301s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:18.292896529 +0000 UTC m=+1100.373482756" watchObservedRunningTime="2026-10-30 18:55:18.34193849 +0000 UTC m=+1100.422524697"
Oct 30 18:55:20.316372 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:20.316337    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:55:20.317845 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:20.317817    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:55:28.083011 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:55:28.082973    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:55:30.316388 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:30.316344    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:55:40.316714 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:40.316616    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:55:50.315826 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:55:50.315779    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:55:58.100623 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:55:58.100580    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:56:00.316153 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:00.316117    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:56:10.316957 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:10.316904    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:56:15.961891 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:15.961855    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"]
Oct 30 18:56:15.965732 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:15.965709    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.041910 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.041880    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"]
Oct 30 18:56:16.126677 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.126645    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233fa757-125e-4977-9c25-233988590a43-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.126861 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.126795    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj94r\" (UniqueName: \"kubernetes.io/projected/233fa757-125e-4977-9c25-233988590a43-kube-api-access-kj94r\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.127759 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.127729    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-x4x8q\""
Oct 30 18:56:16.127759 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.127740    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Oct 30 18:56:16.127759 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.127751    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-2nbtr\""
Oct 30 18:56:16.127759 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.127740    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Oct 30 18:56:16.127939 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.127798    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Oct 30 18:56:16.227559 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.227491    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj94r\" (UniqueName: \"kubernetes.io/projected/233fa757-125e-4977-9c25-233988590a43-kube-api-access-kj94r\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.227559 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.227559    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233fa757-125e-4977-9c25-233988590a43-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.230047 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.230017    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233fa757-125e-4977-9c25-233988590a43-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.283411 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.283382    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj94r\" (UniqueName: \"kubernetes.io/projected/233fa757-125e-4977-9c25-233988590a43-kube-api-access-kj94r\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.579872 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.579794    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:16.956094 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:16.956053    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"]
Oct 30 18:56:16.966027 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:56:16.965996    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233fa757_125e_4977_9c25_233988590a43.slice/crio-dbdf12e51bcdfbe94ae20bff7e2d5be7db33b9f8016c09147491d09e2624bc63 WatchSource:0}: Error finding container dbdf12e51bcdfbe94ae20bff7e2d5be7db33b9f8016c09147491d09e2624bc63: Status 404 returned error can't find the container with id dbdf12e51bcdfbe94ae20bff7e2d5be7db33b9f8016c09147491d09e2624bc63
Oct 30 18:56:17.422429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:17.422378    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" event={"ID":"233fa757-125e-4977-9c25-233988590a43","Type":"ContainerStarted","Data":"a6df438d8bc3d705119864e36995792d66df1536e9fbec778ffe3021500d8c87"}
Oct 30 18:56:17.422638 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:17.422515    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" event={"ID":"233fa757-125e-4977-9c25-233988590a43","Type":"ContainerStarted","Data":"dbdf12e51bcdfbe94ae20bff7e2d5be7db33b9f8016c09147491d09e2624bc63"}
Oct 30 18:56:17.536698 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:17.536640    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podStartSLOduration=2.536623805 podStartE2EDuration="2.536623805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:17.534645719 +0000 UTC m=+1159.615231925" watchObservedRunningTime="2026-10-30 18:56:17.536623805 +0000 UTC m=+1159.617210013"
Oct 30 18:56:18.427517 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:18.427485    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:56:20.316838 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:20.316795    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:56:28.084016 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:56:28.083982    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:56:30.316104 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:30.316062    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:56:40.325865 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:40.325830    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:56:40.332993 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:40.332966    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:56:48.117779 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:48.117737    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"]
Oct 30 18:56:48.119568 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:48.119540    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"]
Oct 30 18:56:49.435699 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:49.435665    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:56:54.592047 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:54.592014    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"]
Oct 30 18:56:54.593941 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:54.593903    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:56:55.597364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:55.597307    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:56:58.083508 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:56:58.083443    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:56:58.242367 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:58.242335    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:56:58.244401 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:56:58.244377    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 18:57:03.139227 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:03.139190    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" containerID="cri-o://809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5" gracePeriod=45
Oct 30 18:57:03.139772 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:03.139446    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main" containerID="cri-o://4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa" gracePeriod=45
Oct 30 18:57:03.326984 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:03.326942    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 18:57:03.995897 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:03.995872    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:57:04.186713 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.186678    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kube-api-access-t9t4h\" (UniqueName: \"kubernetes.io/projected/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kube-api-access-t9t4h\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.186727    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-dshm\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.186789    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-model-cache\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.186818    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-home\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-home\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.186851    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tls-certs\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.186937    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-cabundle-cert\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.186975    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tmp-dir\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.187029    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kserve-provision-location\") pod \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\" (UID: \"d3262dee-f035-4c5e-b9a2-5eaf83dffb84\") "
Oct 30 18:57:04.187196 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.187107    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-model-cache" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:04.187586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.187375    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-home" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:04.187586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.187411    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.187586 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.187453    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-cabundle-cert" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:57:04.189570 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.189538    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-dshm" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:04.189817 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.189796    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kube-api-access-t9t4h" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "kube-api-access-t9t4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:04.189890 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.189872    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tls-certs" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:04.198785 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.198762    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tmp-dir" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:04.218297 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.218249    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kserve-provision-location" pod "d3262dee-f035-4c5e-b9a2-5eaf83dffb84" (UID: "d3262dee-f035-4c5e-b9a2-5eaf83dffb84"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:57:04.288107 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.288072    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.288107 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.288099    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.288275 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.288114    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.288275 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.288127    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9t4h\" (UniqueName: \"kubernetes.io/projected/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-kube-api-access-t9t4h\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.288275 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.288136    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.288275 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.288145    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.288275 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.288153    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3262dee-f035-4c5e-b9a2-5eaf83dffb84-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:04.594013 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.593941    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:04.659447 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.659413    5025 generic.go:356] "Generic (PLEG): container finished" podID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerID="809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5" exitCode=0
Oct 30 18:57:04.659592 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.659454    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" event={"ID":"d3262dee-f035-4c5e-b9a2-5eaf83dffb84","Type":"ContainerDied","Data":"809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5"}
Oct 30 18:57:04.659592 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.659470    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"
Oct 30 18:57:04.659592 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.659494    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc" event={"ID":"d3262dee-f035-4c5e-b9a2-5eaf83dffb84","Type":"ContainerDied","Data":"66ad6da919345e09e9959575febd4ffa61798744caaf43beecbf3240f7cfa894"}
Oct 30 18:57:04.659592 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.659516    5025 scope.go:122] "RemoveContainer" containerID="809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5"
Oct 30 18:57:04.669269 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.669247    5025 scope.go:122] "RemoveContainer" containerID="57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514"
Oct 30 18:57:04.676679 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.676663    5025 scope.go:122] "RemoveContainer" containerID="809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5"
Oct 30 18:57:04.676928 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:57:04.676910    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5\": container with ID starting with 809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5 not found: ID does not exist" containerID="809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5"
Oct 30 18:57:04.676989 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.676936    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5"} err="failed to get container status \"809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5\": rpc error: code = NotFound desc = could not find container \"809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5\": container with ID starting with 809b716821a92106f06481ff045196446247e4a1866788be906ac45b5a1857e5 not found: ID does not exist"
Oct 30 18:57:04.676989 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.676952    5025 scope.go:122] "RemoveContainer" containerID="57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514"
Oct 30 18:57:04.677194 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:57:04.677175    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514\": container with ID starting with 57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514 not found: ID does not exist" containerID="57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514"
Oct 30 18:57:04.677238 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.677200    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514"} err="failed to get container status \"57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514\": rpc error: code = NotFound desc = could not find container \"57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514\": container with ID starting with 57288a20e58b53544df2f012afdd14606a6864796b8534895770bccdbc0df514 not found: ID does not exist"
Oct 30 18:57:04.820145 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.820111    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"]
Oct 30 18:57:04.867486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:04.867461    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpu-accelerator-preset-ser-afc820a4-kserve-f6866ff9c-5mzlc"]
Oct 30 18:57:05.597551 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:05.597516    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:06.057189 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:06.057158    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" path="/var/lib/kubelet/pods/d3262dee-f035-4c5e-b9a2-5eaf83dffb84/volumes"
Oct 30 18:57:09.613474 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:09.613382    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main" containerID="cri-o://a6df438d8bc3d705119864e36995792d66df1536e9fbec778ffe3021500d8c87" gracePeriod=45
Oct 30 18:57:14.593956 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:14.593915    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:15.597347 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:15.597288    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:15.597814 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:15.597464    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:57:24.594080 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:24.594042    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:25.597449 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:25.597404    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:28.081756 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:57:28.081715    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:57:33.399084 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.399020    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:57:33.541967 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.541937    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791fd137-c4e7-49cb-954d-c3c447d3b156-kube-api-access-xs296\" (UniqueName: \"kubernetes.io/projected/791fd137-c4e7-49cb-954d-c3c447d3b156-kube-api-access-xs296\") pod \"791fd137-c4e7-49cb-954d-c3c447d3b156\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") "
Oct 30 18:57:33.542151 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.542044    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/791fd137-c4e7-49cb-954d-c3c447d3b156-tls-certs\" (UniqueName: \"kubernetes.io/secret/791fd137-c4e7-49cb-954d-c3c447d3b156-tls-certs\") pod \"791fd137-c4e7-49cb-954d-c3c447d3b156\" (UID: \"791fd137-c4e7-49cb-954d-c3c447d3b156\") "
Oct 30 18:57:33.544086 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.544054    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791fd137-c4e7-49cb-954d-c3c447d3b156-kube-api-access-xs296" pod "791fd137-c4e7-49cb-954d-c3c447d3b156" (UID: "791fd137-c4e7-49cb-954d-c3c447d3b156"). InnerVolumeSpecName "kube-api-access-xs296". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:33.544199 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.544062    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791fd137-c4e7-49cb-954d-c3c447d3b156-tls-certs" pod "791fd137-c4e7-49cb-954d-c3c447d3b156" (UID: "791fd137-c4e7-49cb-954d-c3c447d3b156"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:33.642887 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.642848    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs296\" (UniqueName: \"kubernetes.io/projected/791fd137-c4e7-49cb-954d-c3c447d3b156-kube-api-access-xs296\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:33.642887 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.642881    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791fd137-c4e7-49cb-954d-c3c447d3b156-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:33.801452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.801364    5025 generic.go:356] "Generic (PLEG): container finished" podID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerID="4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa" exitCode=0
Oct 30 18:57:33.801452 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.801412    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"
Oct 30 18:57:33.801686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.801452    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" event={"ID":"791fd137-c4e7-49cb-954d-c3c447d3b156","Type":"ContainerDied","Data":"4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa"}
Oct 30 18:57:33.801686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.801512    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9" event={"ID":"791fd137-c4e7-49cb-954d-c3c447d3b156","Type":"ContainerDied","Data":"f2bb4acc3db66c1ba7ae79b7c42d7c35931fe1107c4a371d98ffaaa1fba8eeec"}
Oct 30 18:57:33.801686 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.801542    5025 scope.go:122] "RemoveContainer" containerID="4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa"
Oct 30 18:57:33.810922 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.810905    5025 scope.go:122] "RemoveContainer" containerID="4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa"
Oct 30 18:57:33.811172 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:57:33.811153    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa\": container with ID starting with 4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa not found: ID does not exist" containerID="4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa"
Oct 30 18:57:33.811221 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.811183    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa"} err="failed to get container status \"4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa\": rpc error: code = NotFound desc = could not find container \"4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa\": container with ID starting with 4e50265dc820a0181379438721a192724492031824fb041cb639e049536467fa not found: ID does not exist"
Oct 30 18:57:33.981882 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:33.981845    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"]
Oct 30 18:57:34.029206 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:34.029173    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-cpu-accelerator-preset-serves-inference-a4a93d2c/test-cpb25a14c9ba423627edeb5388343bd353-kserve-router-scheqlwr9"]
Oct 30 18:57:34.056690 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:34.056616    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" path="/var/lib/kubelet/pods/791fd137-c4e7-49cb-954d-c3c447d3b156/volumes"
Oct 30 18:57:34.594750 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:34.594713    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:57:39.830369 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:39.830316    5025 generic.go:356] "Generic (PLEG): container finished" podID="233fa757-125e-4977-9c25-233988590a43" containerID="a6df438d8bc3d705119864e36995792d66df1536e9fbec778ffe3021500d8c87" exitCode=0
Oct 30 18:57:39.830813 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:39.830388    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" event={"ID":"233fa757-125e-4977-9c25-233988590a43","Type":"ContainerDied","Data":"a6df438d8bc3d705119864e36995792d66df1536e9fbec778ffe3021500d8c87"}
Oct 30 18:57:39.874122 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:39.874098    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:57:39.999238 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:39.999175    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/233fa757-125e-4977-9c25-233988590a43-kube-api-access-kj94r\" (UniqueName: \"kubernetes.io/projected/233fa757-125e-4977-9c25-233988590a43-kube-api-access-kj94r\") pod \"233fa757-125e-4977-9c25-233988590a43\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") "
Oct 30 18:57:39.999381 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:39.999294    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/233fa757-125e-4977-9c25-233988590a43-tls-certs\" (UniqueName: \"kubernetes.io/secret/233fa757-125e-4977-9c25-233988590a43-tls-certs\") pod \"233fa757-125e-4977-9c25-233988590a43\" (UID: \"233fa757-125e-4977-9c25-233988590a43\") "
Oct 30 18:57:40.001476 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.001426    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/233fa757-125e-4977-9c25-233988590a43-tls-certs" pod "233fa757-125e-4977-9c25-233988590a43" (UID: "233fa757-125e-4977-9c25-233988590a43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:57:40.001620 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.001600    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/233fa757-125e-4977-9c25-233988590a43-kube-api-access-kj94r" pod "233fa757-125e-4977-9c25-233988590a43" (UID: "233fa757-125e-4977-9c25-233988590a43"). InnerVolumeSpecName "kube-api-access-kj94r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:57:40.100781 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.100752    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/233fa757-125e-4977-9c25-233988590a43-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:40.100781 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.100778    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj94r\" (UniqueName: \"kubernetes.io/projected/233fa757-125e-4977-9c25-233988590a43-kube-api-access-kj94r\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:57:40.835897 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.835861    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6" event={"ID":"233fa757-125e-4977-9c25-233988590a43","Type":"ContainerDied","Data":"dbdf12e51bcdfbe94ae20bff7e2d5be7db33b9f8016c09147491d09e2624bc63"}
Oct 30 18:57:40.836354 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.835903    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"
Oct 30 18:57:40.836354 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.835920    5025 scope.go:122] "RemoveContainer" containerID="a6df438d8bc3d705119864e36995792d66df1536e9fbec778ffe3021500d8c87"
Oct 30 18:57:40.996918 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:40.996886    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"]
Oct 30 18:57:41.044014 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:41.043984    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-84865b66smv6"]
Oct 30 18:57:42.056039 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:42.056009    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="233fa757-125e-4977-9c25-233988590a43" path="/var/lib/kubelet/pods/233fa757-125e-4977-9c25-233988590a43/volumes"
Oct 30 18:57:45.728697 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.728663    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"]
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729035    5025 cpu_manager.go:414] "containerMap: removing container" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main"
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729046    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main"
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729054    5025 cpu_manager.go:414] "containerMap: removing container" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main"
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729058    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main"
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729070    5025 cpu_manager.go:414] "containerMap: removing container" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="storage-initializer"
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729075    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="storage-initializer"
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729086    5025 cpu_manager.go:414] "containerMap: removing container" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main"
Oct 30 18:57:45.729098 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729090    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main"
Oct 30 18:57:45.729360 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729159    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3262dee-f035-4c5e-b9a2-5eaf83dffb84" containerName="main"
Oct 30 18:57:45.729360 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729166    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="233fa757-125e-4977-9c25-233988590a43" containerName="main"
Oct 30 18:57:45.729360 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.729175    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="791fd137-c4e7-49cb-954d-c3c447d3b156" containerName="main"
Oct 30 18:57:45.732332 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.732300    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.801197 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.801168    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Oct 30 18:57:45.830486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.830458    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Oct 30 18:57:45.830639 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.830498    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Oct 30 18:57:45.830639 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.830564    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-b4mrj\""
Oct 30 18:57:45.832885 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.832863    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"]
Oct 30 18:57:45.857201 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.857181    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.857303 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.857233    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtn42\" (UniqueName: \"kubernetes.io/projected/bd4e7501-9cfb-445c-bf9d-19a2261b8187-kube-api-access-wtn42\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.857390 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.857307    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.857444 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.857398    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.857481 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.857455    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.857481 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.857474    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.957955 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.957924    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtn42\" (UniqueName: \"kubernetes.io/projected/bd4e7501-9cfb-445c-bf9d-19a2261b8187-kube-api-access-wtn42\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.957984    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.958017    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.958054    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.958080    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958134 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.958125    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958502 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.958478    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-home\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958502 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.958497    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-model-cache\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.958610 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.958532    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tmp-dir\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.960417 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.960396    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-dshm\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:45.960574 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:45.960557    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tls-certs\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:46.015585 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:46.015529    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtn42\" (UniqueName: \"kubernetes.io/projected/bd4e7501-9cfb-445c-bf9d-19a2261b8187-kube-api-access-wtn42\") pod \"fc-smoke-test-kserve-7d8f9d9664-d2c4n\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:46.045436 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:46.045410    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:46.427750 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:46.427721    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"]
Oct 30 18:57:46.433511 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:57:46.433481    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd4e7501_9cfb_445c_bf9d_19a2261b8187.slice/crio-cfac95eb28b5ed7b8b0d7cd64fafd29a1e03341eee5e031cd157d2d4441a8c89 WatchSource:0}: Error finding container cfac95eb28b5ed7b8b0d7cd64fafd29a1e03341eee5e031cd157d2d4441a8c89: Status 404 returned error can't find the container with id cfac95eb28b5ed7b8b0d7cd64fafd29a1e03341eee5e031cd157d2d4441a8c89
Oct 30 18:57:46.435312 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:46.435292    5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 18:57:46.866084 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:46.865996    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" event={"ID":"bd4e7501-9cfb-445c-bf9d-19a2261b8187","Type":"ContainerStarted","Data":"cfac95eb28b5ed7b8b0d7cd64fafd29a1e03341eee5e031cd157d2d4441a8c89"}
Oct 30 18:57:49.904345 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.883239    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"]
Oct 30 18:57:49.927736 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.927690    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:49.946355 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.946298    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"]
Oct 30 18:57:49.951993 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.949908    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4jg49-j7m87"]
Oct 30 18:57:49.954552 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.954528    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.028398 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.995916    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Oct 30 18:57:50.028398 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.996735    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Oct 30 18:57:50.028398 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.996824    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Oct 30 18:57:50.028398 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.996915    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-fbc2j\""
Oct 30 18:57:50.028398 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:49.997012    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-klbsp\""
Oct 30 18:57:50.046113 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.046078    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4jg49-j7m87"]
Oct 30 18:57:50.085713 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.085662    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-xm628\""
Oct 30 18:57:50.143755 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.143721    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-service-ca-bundle\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.144272 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.144253    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6855\" (UniqueName: \"kubernetes.io/projected/9fbba4f8-d6db-40d6-a566-08e80f748611-kube-api-access-t6855\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.144499 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.144480    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.144635 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.144622    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbba4f8-d6db-40d6-a566-08e80f748611-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.144768 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.144757    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-trusted-ca-bundle\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.144895 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.144883    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.145008 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.144998    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e53f6b3-c385-4981-916d-733990f471b1-archives-path\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.145127 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.145116    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqtb\" (UniqueName: \"kubernetes.io/projected/7e53f6b3-c385-4981-916d-733990f471b1-kube-api-access-dwqtb\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245798    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-trusted-ca-bundle\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245840    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245861    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e53f6b3-c385-4981-916d-733990f471b1-archives-path\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245877    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqtb\" (UniqueName: \"kubernetes.io/projected/7e53f6b3-c385-4981-916d-733990f471b1-kube-api-access-dwqtb\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245913    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-service-ca-bundle\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245930    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6855\" (UniqueName: \"kubernetes.io/projected/9fbba4f8-d6db-40d6-a566-08e80f748611-kube-api-access-t6855\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245973    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.246429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.245988    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbba4f8-d6db-40d6-a566-08e80f748611-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.248862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.247436    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-trusted-ca-bundle\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.248862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.247648    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.248862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.247829    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e53f6b3-c385-4981-916d-733990f471b1-archives-path\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.248862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.248491    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-service-ca-bundle\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.248862 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.248834    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.249941 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.249918    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbba4f8-d6db-40d6-a566-08e80f748611-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.309040 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.309005    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqtb\" (UniqueName: \"kubernetes.io/projected/7e53f6b3-c385-4981-916d-733990f471b1-kube-api-access-dwqtb\") pod \"periodic-gathering-4jg49-j7m87\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") " pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.336347 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.334097    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6855\" (UniqueName: \"kubernetes.io/projected/9fbba4f8-d6db-40d6-a566-08e80f748611-kube-api-access-t6855\") pod \"scheduler-custom-template-test-kserve-router-scheduler-bd57s62w\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.350122 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.350083    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:57:50.641372 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.587523    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:50.942363 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:50.923716    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4jg49-j7m87"]
Oct 30 18:57:50.947565 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:57:50.947530    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e53f6b3_c385_4981_916d_733990f471b1.slice/crio-0926a13b03b831dd5a1ff49a75f6c62b8634b05bdafc017649ec56e53491cfde WatchSource:0}: Error finding container 0926a13b03b831dd5a1ff49a75f6c62b8634b05bdafc017649ec56e53491cfde: Status 404 returned error can't find the container with id 0926a13b03b831dd5a1ff49a75f6c62b8634b05bdafc017649ec56e53491cfde
Oct 30 18:57:51.200504 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:51.200466    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"]
Oct 30 18:57:51.975347 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:51.969892    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" event={"ID":"9fbba4f8-d6db-40d6-a566-08e80f748611","Type":"ContainerStarted","Data":"70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373"}
Oct 30 18:57:51.975347 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:51.969967    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" event={"ID":"9fbba4f8-d6db-40d6-a566-08e80f748611","Type":"ContainerStarted","Data":"74f5098376de0161ed5e236c5839cda40cde3a1015fc151bac114e3f0ff32dbe"}
Oct 30 18:57:51.988364 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:51.975957    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:57:51.993070 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:51.993026    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4jg49-j7m87" event={"ID":"7e53f6b3-c385-4981-916d-733990f471b1","Type":"ContainerStarted","Data":"55ebc912feb81223e2522b7945089bdf5c44958f115795e7d8c3f086c013b1a7"}
Oct 30 18:57:51.993291 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:51.993263    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4jg49-j7m87" event={"ID":"7e53f6b3-c385-4981-916d-733990f471b1","Type":"ContainerStarted","Data":"0926a13b03b831dd5a1ff49a75f6c62b8634b05bdafc017649ec56e53491cfde"}
Oct 30 18:57:52.246691 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:52.246586    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4jg49-j7m87" podStartSLOduration=3.246571071 podStartE2EDuration="3.246571071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:52.246116783 +0000 UTC m=+1254.326703009" watchObservedRunningTime="2026-10-30 18:57:52.246571071 +0000 UTC m=+1254.327157278"
Oct 30 18:57:52.247430 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:52.247392    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" podStartSLOduration=3.247381344 podStartE2EDuration="3.247381344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:52.121697868 +0000 UTC m=+1254.202284104" watchObservedRunningTime="2026-10-30 18:57:52.247381344 +0000 UTC m=+1254.327967551"
Oct 30 18:57:53.046885 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:53.046842    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" event={"ID":"bd4e7501-9cfb-445c-bf9d-19a2261b8187","Type":"ContainerStarted","Data":"0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14"}
Oct 30 18:57:54.045716 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:54.045676    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:56.045968 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:56.045931    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:56.059361 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:56.059313    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:56.072824 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:56.072801    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:57:56.172562 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:57:56.172519    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" podStartSLOduration=5.066513374 podStartE2EDuration="11.172505176s" totalImagesPullingTime="6.105991802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:53.165811784 +0000 UTC m=+1255.246398016" watchObservedRunningTime="2026-10-30 18:57:56.172505176 +0000 UTC m=+1258.253091395"
Oct 30 18:57:58.082804 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:57:58.082769    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:58:24.061986 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:24.061953    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:58:28.082573 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:58:28.082534    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:58:29.366606 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.366569    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"]
Oct 30 18:58:29.376134 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:58:29.376089    5025 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14" cmd=["/bin/sleep","15"]
Oct 30 18:58:29.376307 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.376153    5025 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 18:58:29.376307 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:58:29.376171    5025 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" podUID="bd4e7501-9cfb-445c-bf9d-19a2261b8187" containerName="main" containerID="cri-o://0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14"
Oct 30 18:58:29.376307 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.376222    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" podUID="bd4e7501-9cfb-445c-bf9d-19a2261b8187" containerName="main" containerID="cri-o://0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14" gracePeriod=60
Oct 30 18:58:29.732191 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.732169    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:58:29.811143 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811117    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-home\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-home\") pod \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") "
Oct 30 18:58:29.811143 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811148    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-model-cache\") pod \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") "
Oct 30 18:58:29.811371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811190    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tmp-dir\") pod \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") "
Oct 30 18:58:29.811371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811217    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-dshm\") pod \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") "
Oct 30 18:58:29.811371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811245    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tls-certs\") pod \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") "
Oct 30 18:58:29.811517 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811380    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd4e7501-9cfb-445c-bf9d-19a2261b8187-kube-api-access-wtn42\" (UniqueName: \"kubernetes.io/projected/bd4e7501-9cfb-445c-bf9d-19a2261b8187-kube-api-access-wtn42\") pod \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\" (UID: \"bd4e7501-9cfb-445c-bf9d-19a2261b8187\") "
Oct 30 18:58:29.811517 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811398    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-home" pod "bd4e7501-9cfb-445c-bf9d-19a2261b8187" (UID: "bd4e7501-9cfb-445c-bf9d-19a2261b8187"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:29.811517 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811430    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-model-cache" pod "bd4e7501-9cfb-445c-bf9d-19a2261b8187" (UID: "bd4e7501-9cfb-445c-bf9d-19a2261b8187"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:29.811673 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811519    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tmp-dir" pod "bd4e7501-9cfb-445c-bf9d-19a2261b8187" (UID: "bd4e7501-9cfb-445c-bf9d-19a2261b8187"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:29.811673 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811633    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:29.811673 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811645    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:29.811673 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.811655    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:29.813580 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.813550    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tls-certs" pod "bd4e7501-9cfb-445c-bf9d-19a2261b8187" (UID: "bd4e7501-9cfb-445c-bf9d-19a2261b8187"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:58:29.813667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.813629    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-dshm" pod "bd4e7501-9cfb-445c-bf9d-19a2261b8187" (UID: "bd4e7501-9cfb-445c-bf9d-19a2261b8187"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:58:29.813667 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.813644    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd4e7501-9cfb-445c-bf9d-19a2261b8187-kube-api-access-wtn42" pod "bd4e7501-9cfb-445c-bf9d-19a2261b8187" (UID: "bd4e7501-9cfb-445c-bf9d-19a2261b8187"). InnerVolumeSpecName "kube-api-access-wtn42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:58:29.912725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.912645    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd4e7501-9cfb-445c-bf9d-19a2261b8187-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:29.912725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.912678    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtn42\" (UniqueName: \"kubernetes.io/projected/bd4e7501-9cfb-445c-bf9d-19a2261b8187-kube-api-access-wtn42\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:29.912725 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:29.912689    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd4e7501-9cfb-445c-bf9d-19a2261b8187-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:58:30.226771 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.226689    5025 generic.go:356] "Generic (PLEG): container finished" podID="bd4e7501-9cfb-445c-bf9d-19a2261b8187" containerID="0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14" exitCode=0
Oct 30 18:58:30.226771 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.226749    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"
Oct 30 18:58:30.226960 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.226769    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" event={"ID":"bd4e7501-9cfb-445c-bf9d-19a2261b8187","Type":"ContainerDied","Data":"0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14"}
Oct 30 18:58:30.226960 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.226823    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n" event={"ID":"bd4e7501-9cfb-445c-bf9d-19a2261b8187","Type":"ContainerDied","Data":"cfac95eb28b5ed7b8b0d7cd64fafd29a1e03341eee5e031cd157d2d4441a8c89"}
Oct 30 18:58:30.226960 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.226844    5025 scope.go:122] "RemoveContainer" containerID="0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14"
Oct 30 18:58:30.235781 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.235762    5025 scope.go:122] "RemoveContainer" containerID="0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14"
Oct 30 18:58:30.236052 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:58:30.236030    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14\": container with ID starting with 0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14 not found: ID does not exist" containerID="0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14"
Oct 30 18:58:30.236104 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.236059    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14"} err="failed to get container status \"0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14\": rpc error: code = NotFound desc = could not find container \"0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14\": container with ID starting with 0814ed7293349450aa3e9c4c369e2f7e81ec46aeee03cce805fe3a26348f8d14 not found: ID does not exist"
Oct 30 18:58:30.388125 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.388092    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"]
Oct 30 18:58:30.435607 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.435576    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7d8f9d9664-d2c4n"]
Oct 30 18:58:30.539221 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:30.539145    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"]
Oct 30 18:58:32.056877 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:32.056837    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd4e7501-9cfb-445c-bf9d-19a2261b8187" path="/var/lib/kubelet/pods/bd4e7501-9cfb-445c-bf9d-19a2261b8187/volumes"
Oct 30 18:58:34.053411 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:34.053368    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:58:44.053592 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:44.053504    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:58:45.558999 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:45.558932    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main" containerID="cri-o://70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373" gracePeriod=45
Oct 30 18:58:54.053990 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:54.053947    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:58:54.056670 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:58:54.056646    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:58:58.083807 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:58:58.083771    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:59:04.053954 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:04.053912    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:59:14.053984 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:14.053944    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 18:59:15.816762 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.816740    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:59:15.932570 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.932539    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbba4f8-d6db-40d6-a566-08e80f748611-tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbba4f8-d6db-40d6-a566-08e80f748611-tls-certs\") pod \"9fbba4f8-d6db-40d6-a566-08e80f748611\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") "
Oct 30 18:59:15.932779 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.932585    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-tokenizer-cache\") pod \"9fbba4f8-d6db-40d6-a566-08e80f748611\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") "
Oct 30 18:59:15.932779 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.932663    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbba4f8-d6db-40d6-a566-08e80f748611-kube-api-access-t6855\" (UniqueName: \"kubernetes.io/projected/9fbba4f8-d6db-40d6-a566-08e80f748611-kube-api-access-t6855\") pod \"9fbba4f8-d6db-40d6-a566-08e80f748611\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") "
Oct 30 18:59:15.932779 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.932708    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-cachi2-cache\") pod \"9fbba4f8-d6db-40d6-a566-08e80f748611\" (UID: \"9fbba4f8-d6db-40d6-a566-08e80f748611\") "
Oct 30 18:59:15.933128 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.933099    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-tokenizer-cache" pod "9fbba4f8-d6db-40d6-a566-08e80f748611" (UID: "9fbba4f8-d6db-40d6-a566-08e80f748611"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:15.933300 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.933212    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-cachi2-cache" pod "9fbba4f8-d6db-40d6-a566-08e80f748611" (UID: "9fbba4f8-d6db-40d6-a566-08e80f748611"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:15.935340 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.935293    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbba4f8-d6db-40d6-a566-08e80f748611-kube-api-access-t6855" pod "9fbba4f8-d6db-40d6-a566-08e80f748611" (UID: "9fbba4f8-d6db-40d6-a566-08e80f748611"). InnerVolumeSpecName "kube-api-access-t6855". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:15.935448 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:15.935420    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbba4f8-d6db-40d6-a566-08e80f748611-tls-certs" pod "9fbba4f8-d6db-40d6-a566-08e80f748611" (UID: "9fbba4f8-d6db-40d6-a566-08e80f748611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 18:59:16.034439 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.034405    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fbba4f8-d6db-40d6-a566-08e80f748611-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:16.034439 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.034434    5025 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-tokenizer-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:16.034625 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.034449    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6855\" (UniqueName: \"kubernetes.io/projected/9fbba4f8-d6db-40d6-a566-08e80f748611-kube-api-access-t6855\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:16.034625 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.034459    5025 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/9fbba4f8-d6db-40d6-a566-08e80f748611-cachi2-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:16.442439 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.442404    5025 generic.go:356] "Generic (PLEG): container finished" podID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerID="70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373" exitCode=0
Oct 30 18:59:16.442587 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.442456    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"
Oct 30 18:59:16.442587 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.442462    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" event={"ID":"9fbba4f8-d6db-40d6-a566-08e80f748611","Type":"ContainerDied","Data":"70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373"}
Oct 30 18:59:16.442587 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.442527    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w" event={"ID":"9fbba4f8-d6db-40d6-a566-08e80f748611","Type":"ContainerDied","Data":"74f5098376de0161ed5e236c5839cda40cde3a1015fc151bac114e3f0ff32dbe"}
Oct 30 18:59:16.442587 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.442556    5025 scope.go:122] "RemoveContainer" containerID="70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373"
Oct 30 18:59:16.451111 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.451091    5025 scope.go:122] "RemoveContainer" containerID="70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373"
Oct 30 18:59:16.451389 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:59:16.451367    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373\": container with ID starting with 70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373 not found: ID does not exist" containerID="70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373"
Oct 30 18:59:16.451442 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.451398    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373"} err="failed to get container status \"70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373\": rpc error: code = NotFound desc = could not find container \"70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373\": container with ID starting with 70a33e14324cdcc3776d74d472ed9f810acf1d811b782d19a92f6ae57daf9373 not found: ID does not exist"
Oct 30 18:59:16.602084 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.602053    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"]
Oct 30 18:59:16.649680 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:16.649649    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-bd57s62w"]
Oct 30 18:59:18.062220 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:18.062187    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" path="/var/lib/kubelet/pods/9fbba4f8-d6db-40d6-a566-08e80f748611/volumes"
Oct 30 18:59:25.484998 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:25.484963    5025 generic.go:356] "Generic (PLEG): container finished" podID="7e53f6b3-c385-4981-916d-733990f471b1" containerID="55ebc912feb81223e2522b7945089bdf5c44958f115795e7d8c3f086c013b1a7" exitCode=0
Oct 30 18:59:25.485499 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:25.485044    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4jg49-j7m87" event={"ID":"7e53f6b3-c385-4981-916d-733990f471b1","Type":"ContainerDied","Data":"55ebc912feb81223e2522b7945089bdf5c44958f115795e7d8c3f086c013b1a7"}
Oct 30 18:59:26.632398 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.632372    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:59:26.732813 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.732774    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-trusted-ca-bundle\") pod \"7e53f6b3-c385-4981-916d-733990f471b1\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") "
Oct 30 18:59:26.732974 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.732835    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-service-ca-bundle\") pod \"7e53f6b3-c385-4981-916d-733990f471b1\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") "
Oct 30 18:59:26.732974 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.732877    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e53f6b3-c385-4981-916d-733990f471b1-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e53f6b3-c385-4981-916d-733990f471b1-archives-path\") pod \"7e53f6b3-c385-4981-916d-733990f471b1\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") "
Oct 30 18:59:26.732974 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.732935    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e53f6b3-c385-4981-916d-733990f471b1-kube-api-access-dwqtb\" (UniqueName: \"kubernetes.io/projected/7e53f6b3-c385-4981-916d-733990f471b1-kube-api-access-dwqtb\") pod \"7e53f6b3-c385-4981-916d-733990f471b1\" (UID: \"7e53f6b3-c385-4981-916d-733990f471b1\") "
Oct 30 18:59:26.733170 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.733143    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-trusted-ca-bundle" pod "7e53f6b3-c385-4981-916d-733990f471b1" (UID: "7e53f6b3-c385-4981-916d-733990f471b1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:26.733284 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.733261    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-service-ca-bundle" pod "7e53f6b3-c385-4981-916d-733990f471b1" (UID: "7e53f6b3-c385-4981-916d-733990f471b1"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 18:59:26.733371 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.733259    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e53f6b3-c385-4981-916d-733990f471b1-archives-path" pod "7e53f6b3-c385-4981-916d-733990f471b1" (UID: "7e53f6b3-c385-4981-916d-733990f471b1"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 18:59:26.735133 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.735105    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e53f6b3-c385-4981-916d-733990f471b1-kube-api-access-dwqtb" pod "7e53f6b3-c385-4981-916d-733990f471b1" (UID: "7e53f6b3-c385-4981-916d-733990f471b1"). InnerVolumeSpecName "kube-api-access-dwqtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 18:59:26.834165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.834126    5025 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-trusted-ca-bundle\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:26.834165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.834158    5025 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7e53f6b3-c385-4981-916d-733990f471b1-service-ca-bundle\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:26.834165 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.834168    5025 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/7e53f6b3-c385-4981-916d-733990f471b1-archives-path\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:26.834429 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.834177    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqtb\" (UniqueName: \"kubernetes.io/projected/7e53f6b3-c385-4981-916d-733990f471b1-kube-api-access-dwqtb\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 18:59:26.883724 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.883695    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"]
Oct 30 18:59:26.884121 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884106    5025 cpu_manager.go:414] "containerMap: removing container" podUID="7e53f6b3-c385-4981-916d-733990f471b1" containerName="insights-gathering"
Oct 30 18:59:26.884167 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884122    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e53f6b3-c385-4981-916d-733990f471b1" containerName="insights-gathering"
Oct 30 18:59:26.884167 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884130    5025 cpu_manager.go:414] "containerMap: removing container" podUID="bd4e7501-9cfb-445c-bf9d-19a2261b8187" containerName="main"
Oct 30 18:59:26.884167 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884136    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd4e7501-9cfb-445c-bf9d-19a2261b8187" containerName="main"
Oct 30 18:59:26.884261 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884174    5025 cpu_manager.go:414] "containerMap: removing container" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main"
Oct 30 18:59:26.884261 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884179    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main"
Oct 30 18:59:26.884261 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884240    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbba4f8-d6db-40d6-a566-08e80f748611" containerName="main"
Oct 30 18:59:26.884261 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884250    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e53f6b3-c385-4981-916d-733990f471b1" containerName="insights-gathering"
Oct 30 18:59:26.884261 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.884256    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd4e7501-9cfb-445c-bf9d-19a2261b8187" containerName="main"
Oct 30 18:59:26.887727 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.887704    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:26.943116 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.943093    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-bvgsz\""
Oct 30 18:59:26.943260 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.943244    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Oct 30 18:59:26.943315 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.943280    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-jdkxn\""
Oct 30 18:59:26.989657 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.989629    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Oct 30 18:59:26.989793 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.989680    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Oct 30 18:59:26.991923 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:26.991860    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"]
Oct 30 18:59:27.036278 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.036248    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.036439 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.036300    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.036486 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.036454    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.036520 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.036484    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.036520 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.036505    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jnh6\" (UniqueName: \"kubernetes.io/projected/d8c7df5d-4794-4930-9e1b-28499ccce993-kube-api-access-2jnh6\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.036596 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.036574    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7df5d-4794-4930-9e1b-28499ccce993-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.137607 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.137568    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.137607 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.137615    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.137857 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.137636    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jnh6\" (UniqueName: \"kubernetes.io/projected/d8c7df5d-4794-4930-9e1b-28499ccce993-kube-api-access-2jnh6\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.137857 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.137658    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7df5d-4794-4930-9e1b-28499ccce993-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.137857 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.137709    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.137857 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.137749    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.138065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.138014    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.138065 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.138039    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.138142 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.138095    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-home\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.139894 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.139873    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.140064 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.140048    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7df5d-4794-4930-9e1b-28499ccce993-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.193812 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.193789    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jnh6\" (UniqueName: \"kubernetes.io/projected/d8c7df5d-4794-4930-9e1b-28499ccce993-kube-api-access-2jnh6\") pod \"scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.201651 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.201629    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:27.496541 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.496501    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4jg49-j7m87" event={"ID":"7e53f6b3-c385-4981-916d-733990f471b1","Type":"ContainerDied","Data":"0926a13b03b831dd5a1ff49a75f6c62b8634b05bdafc017649ec56e53491cfde"}
Oct 30 18:59:27.496541 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.496525    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4jg49-j7m87"
Oct 30 18:59:27.496735 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.496554    5025 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0926a13b03b831dd5a1ff49a75f6c62b8634b05bdafc017649ec56e53491cfde"
Oct 30 18:59:27.504384 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.504307    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"]
Oct 30 18:59:27.509312 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.509293    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:27.564210 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.564183    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-qnhgf\""
Oct 30 18:59:27.607484 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.607453    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"]
Oct 30 18:59:27.643935 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.643906    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5cf9bb-a540-4b52-82d6-d147fc112377-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:27.644294 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.643951    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mdk\" (UniqueName: \"kubernetes.io/projected/0c5cf9bb-a540-4b52-82d6-d147fc112377-kube-api-access-n7mdk\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:27.659156 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.659131    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"]
Oct 30 18:59:27.665386 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:59:27.665362    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8c7df5d_4794_4930_9e1b_28499ccce993.slice/crio-16ca5c95b57a7737d6501cb16d70b992e33079c068a1cb49657ceb6ee1a386f6 WatchSource:0}: Error finding container 16ca5c95b57a7737d6501cb16d70b992e33079c068a1cb49657ceb6ee1a386f6: Status 404 returned error can't find the container with id 16ca5c95b57a7737d6501cb16d70b992e33079c068a1cb49657ceb6ee1a386f6
Oct 30 18:59:27.744712 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.744680    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5cf9bb-a540-4b52-82d6-d147fc112377-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:27.744873 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.744730    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mdk\" (UniqueName: \"kubernetes.io/projected/0c5cf9bb-a540-4b52-82d6-d147fc112377-kube-api-access-n7mdk\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:27.747386 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.747360    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5cf9bb-a540-4b52-82d6-d147fc112377-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:27.801606 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.801541    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mdk\" (UniqueName: \"kubernetes.io/projected/0c5cf9bb-a540-4b52-82d6-d147fc112377-kube-api-access-n7mdk\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:27.821029 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:27.821006    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:28.082749 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:59:28.082667    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:59:28.202136 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:28.202102    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"]
Oct 30 18:59:28.207516 ip-10-0-129-88 kubenswrapper[5025]: W1030 18:59:28.207492    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5cf9bb_a540_4b52_82d6_d147fc112377.slice/crio-86252c4003f096a39124aafb66094f930889233d8ca28b79d8e44f9537ddbc31 WatchSource:0}: Error finding container 86252c4003f096a39124aafb66094f930889233d8ca28b79d8e44f9537ddbc31: Status 404 returned error can't find the container with id 86252c4003f096a39124aafb66094f930889233d8ca28b79d8e44f9537ddbc31
Oct 30 18:59:28.503092 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:28.503042    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" event={"ID":"0c5cf9bb-a540-4b52-82d6-d147fc112377","Type":"ContainerStarted","Data":"59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51"}
Oct 30 18:59:28.503291 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:28.503252    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 18:59:28.503378 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:28.503294    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" event={"ID":"0c5cf9bb-a540-4b52-82d6-d147fc112377","Type":"ContainerStarted","Data":"86252c4003f096a39124aafb66094f930889233d8ca28b79d8e44f9537ddbc31"}
Oct 30 18:59:28.504386 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:28.504361    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" event={"ID":"d8c7df5d-4794-4930-9e1b-28499ccce993","Type":"ContainerStarted","Data":"16ca5c95b57a7737d6501cb16d70b992e33079c068a1cb49657ceb6ee1a386f6"}
Oct 30 18:59:28.615524 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:28.615470    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" podStartSLOduration=1.6154573989999998 podStartE2EDuration="1.615457399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:28.614834464 +0000 UTC m=+1350.695420672" watchObservedRunningTime="2026-10-30 18:59:28.615457399 +0000 UTC m=+1350.696043607"
Oct 30 18:59:29.514297 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:29.514251    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" event={"ID":"d8c7df5d-4794-4930-9e1b-28499ccce993","Type":"ContainerStarted","Data":"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"}
Oct 30 18:59:29.514692 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:29.514553    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:30.520395 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:30.520362    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" event={"ID":"d8c7df5d-4794-4930-9e1b-28499ccce993","Type":"ContainerStarted","Data":"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"}
Oct 30 18:59:30.635697 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:30.635648    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" podStartSLOduration=2.864811369 podStartE2EDuration="4.635633314s" totalImagesPullingTime="1.770821945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:30.634021959 +0000 UTC m=+1352.714608166" watchObservedRunningTime="2026-10-30 18:59:30.635633314 +0000 UTC m=+1352.716219520"
Oct 30 18:59:31.202403 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:31.202373    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:37.207349 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:37.206936    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:37.238114 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:37.238088    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:37.657367 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:37.657337    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:48.665343 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:48.665285    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 18:59:58.082343 ip-10-0-129-88 kubenswrapper[5025]: E1030 18:59:58.082296    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 18:59:59.516385 ip-10-0-129-88 kubenswrapper[5025]: I1030 18:59:59.516357    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 19:00:07.574128 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:07.574049    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"]
Oct 30 19:00:07.575766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:07.575733    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"]
Oct 30 19:00:07.583784 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:00:07.583745    5025 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31" cmd=["/bin/sleep","15"]
Oct 30 19:00:07.583907 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:07.583804    5025 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:00:07.583907 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:00:07.583817    5025 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="main" containerID="cri-o://f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"
Oct 30 19:00:07.583907 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:07.583853    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="main" containerID="cri-o://f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31" gracePeriod=60
Oct 30 19:00:07.646082 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:07.646045    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8001/health\": dial tcp 10.132.0.57:8001: connect: connection refused"
Oct 30 19:00:07.801616 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:07.801580    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="llm-d-routing-sidecar" containerID="cri-o://823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e" gracePeriod=60
Oct 30 19:00:08.060848 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.060823    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 19:00:08.226252 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226163    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-model-cache\") pod \"d8c7df5d-4794-4930-9e1b-28499ccce993\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") "
Oct 30 19:00:08.226252 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226232    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8c7df5d-4794-4930-9e1b-28499ccce993-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7df5d-4794-4930-9e1b-28499ccce993-tls-certs\") pod \"d8c7df5d-4794-4930-9e1b-28499ccce993\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") "
Oct 30 19:00:08.226524 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226271    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-tmp-dir\") pod \"d8c7df5d-4794-4930-9e1b-28499ccce993\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") "
Oct 30 19:00:08.226524 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226410    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-home\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-home\") pod \"d8c7df5d-4794-4930-9e1b-28499ccce993\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") "
Oct 30 19:00:08.226524 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226461    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-model-cache" pod "d8c7df5d-4794-4930-9e1b-28499ccce993" (UID: "d8c7df5d-4794-4930-9e1b-28499ccce993"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.226524 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226468    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-dshm\") pod \"d8c7df5d-4794-4930-9e1b-28499ccce993\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") "
Oct 30 19:00:08.226719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226569    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8c7df5d-4794-4930-9e1b-28499ccce993-kube-api-access-2jnh6\" (UniqueName: \"kubernetes.io/projected/d8c7df5d-4794-4930-9e1b-28499ccce993-kube-api-access-2jnh6\") pod \"d8c7df5d-4794-4930-9e1b-28499ccce993\" (UID: \"d8c7df5d-4794-4930-9e1b-28499ccce993\") "
Oct 30 19:00:08.226719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226585    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-tmp-dir" pod "d8c7df5d-4794-4930-9e1b-28499ccce993" (UID: "d8c7df5d-4794-4930-9e1b-28499ccce993"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.226719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226676    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-home" pod "d8c7df5d-4794-4930-9e1b-28499ccce993" (UID: "d8c7df5d-4794-4930-9e1b-28499ccce993"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.226929 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226909    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.226986 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226936    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.226986 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.226952    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.228683 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.228651    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8c7df5d-4794-4930-9e1b-28499ccce993-tls-certs" pod "d8c7df5d-4794-4930-9e1b-28499ccce993" (UID: "d8c7df5d-4794-4930-9e1b-28499ccce993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:08.228790 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.228696    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-dshm" pod "d8c7df5d-4794-4930-9e1b-28499ccce993" (UID: "d8c7df5d-4794-4930-9e1b-28499ccce993"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:00:08.228790 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.228704    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8c7df5d-4794-4930-9e1b-28499ccce993-kube-api-access-2jnh6" pod "d8c7df5d-4794-4930-9e1b-28499ccce993" (UID: "d8c7df5d-4794-4930-9e1b-28499ccce993"). InnerVolumeSpecName "kube-api-access-2jnh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:08.328377 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.328347    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8c7df5d-4794-4930-9e1b-28499ccce993-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.328377 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.328375    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jnh6\" (UniqueName: \"kubernetes.io/projected/d8c7df5d-4794-4930-9e1b-28499ccce993-kube-api-access-2jnh6\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.328534 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.328391    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8c7df5d-4794-4930-9e1b-28499ccce993-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:08.794223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.794189    5025 generic.go:356] "Generic (PLEG): container finished" podID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerID="f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31" exitCode=0
Oct 30 19:00:08.794223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.794213    5025 generic.go:356] "Generic (PLEG): container finished" podID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerID="823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e" exitCode=0
Oct 30 19:00:08.794692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.794237    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"
Oct 30 19:00:08.794692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.794271    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" event={"ID":"d8c7df5d-4794-4930-9e1b-28499ccce993","Type":"ContainerDied","Data":"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"}
Oct 30 19:00:08.794692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.794344    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" event={"ID":"d8c7df5d-4794-4930-9e1b-28499ccce993","Type":"ContainerDied","Data":"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"}
Oct 30 19:00:08.794692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.794369    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6" event={"ID":"d8c7df5d-4794-4930-9e1b-28499ccce993","Type":"ContainerDied","Data":"16ca5c95b57a7737d6501cb16d70b992e33079c068a1cb49657ceb6ee1a386f6"}
Oct 30 19:00:08.794692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.794393    5025 scope.go:122] "RemoveContainer" containerID="f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"
Oct 30 19:00:08.804250 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.804171    5025 scope.go:122] "RemoveContainer" containerID="823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"
Oct 30 19:00:08.811906 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.811888    5025 scope.go:122] "RemoveContainer" containerID="f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"
Oct 30 19:00:08.812154 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:00:08.812133    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31\": container with ID starting with f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31 not found: ID does not exist" containerID="f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"
Oct 30 19:00:08.812206 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.812162    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"} err="failed to get container status \"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31\": rpc error: code = NotFound desc = could not find container \"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31\": container with ID starting with f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31 not found: ID does not exist"
Oct 30 19:00:08.812206 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.812180    5025 scope.go:122] "RemoveContainer" containerID="823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"
Oct 30 19:00:08.812422 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:00:08.812403    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e\": container with ID starting with 823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e not found: ID does not exist" containerID="823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"
Oct 30 19:00:08.812501 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.812431    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"} err="failed to get container status \"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e\": rpc error: code = NotFound desc = could not find container \"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e\": container with ID starting with 823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e not found: ID does not exist"
Oct 30 19:00:08.812501 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.812453    5025 scope.go:122] "RemoveContainer" containerID="f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"
Oct 30 19:00:08.812712 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.812694    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31"} err="failed to get container status \"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31\": rpc error: code = NotFound desc = could not find container \"f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31\": container with ID starting with f8b63e903a4ae2bf17bccc0e790326f08bc382daf9855cd4850b12899f100a31 not found: ID does not exist"
Oct 30 19:00:08.812761 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.812714    5025 scope.go:122] "RemoveContainer" containerID="823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"
Oct 30 19:00:08.812938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:08.812918    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e"} err="failed to get container status \"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e\": rpc error: code = NotFound desc = could not find container \"823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e\": container with ID starting with 823a7cbb1423236d62756b43eb8e7648aa453f061e5fdc66774711d7ad487b9e not found: ID does not exist"
Oct 30 19:00:09.001760 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:09.001733    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"]
Oct 30 19:00:09.048762 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:09.048687    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-8468d8f4d9-hmjs6"]
Oct 30 19:00:09.515820 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:09.515772    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:00:10.055936 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:10.055903    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" path="/var/lib/kubelet/pods/d8c7df5d-4794-4930-9e1b-28499ccce993/volumes"
Oct 30 19:00:19.516866 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:19.516825    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:00:22.595053 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:22.594993    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main" containerID="cri-o://59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51" gracePeriod=45
Oct 30 19:00:28.082124 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:00:28.082086    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:00:29.516564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:29.516524    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:00:29.517049 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:29.516630    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 19:00:39.516629 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:39.516589    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:00:49.516012 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:49.515977    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:00:52.849671 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.849645    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 19:00:52.916315 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.916281    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5cf9bb-a540-4b52-82d6-d147fc112377-kube-api-access-n7mdk\" (UniqueName: \"kubernetes.io/projected/0c5cf9bb-a540-4b52-82d6-d147fc112377-kube-api-access-n7mdk\") pod \"0c5cf9bb-a540-4b52-82d6-d147fc112377\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") "
Oct 30 19:00:52.916502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.916356    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c5cf9bb-a540-4b52-82d6-d147fc112377-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5cf9bb-a540-4b52-82d6-d147fc112377-tls-certs\") pod \"0c5cf9bb-a540-4b52-82d6-d147fc112377\" (UID: \"0c5cf9bb-a540-4b52-82d6-d147fc112377\") "
Oct 30 19:00:52.918434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.918402    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c5cf9bb-a540-4b52-82d6-d147fc112377-tls-certs" pod "0c5cf9bb-a540-4b52-82d6-d147fc112377" (UID: "0c5cf9bb-a540-4b52-82d6-d147fc112377"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:00:52.918550 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.918434    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5cf9bb-a540-4b52-82d6-d147fc112377-kube-api-access-n7mdk" pod "0c5cf9bb-a540-4b52-82d6-d147fc112377" (UID: "0c5cf9bb-a540-4b52-82d6-d147fc112377"). InnerVolumeSpecName "kube-api-access-n7mdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:00:52.997740 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.997709    5025 generic.go:356] "Generic (PLEG): container finished" podID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerID="59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51" exitCode=0
Oct 30 19:00:52.997892 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.997766    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" event={"ID":"0c5cf9bb-a540-4b52-82d6-d147fc112377","Type":"ContainerDied","Data":"59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51"}
Oct 30 19:00:52.997892 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.997770    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"
Oct 30 19:00:52.997892 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.997809    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw" event={"ID":"0c5cf9bb-a540-4b52-82d6-d147fc112377","Type":"ContainerDied","Data":"86252c4003f096a39124aafb66094f930889233d8ca28b79d8e44f9537ddbc31"}
Oct 30 19:00:52.997892 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:52.997830    5025 scope.go:122] "RemoveContainer" containerID="59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51"
Oct 30 19:00:53.006667 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:53.006647    5025 scope.go:122] "RemoveContainer" containerID="59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51"
Oct 30 19:00:53.006944 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:00:53.006919    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51\": container with ID starting with 59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51 not found: ID does not exist" containerID="59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51"
Oct 30 19:00:53.007062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:53.006954    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51"} err="failed to get container status \"59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51\": rpc error: code = NotFound desc = could not find container \"59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51\": container with ID starting with 59035d1ba83efb520ba9ac1c288ce84b916a7116692c8b8932ce2f333db9ba51 not found: ID does not exist"
Oct 30 19:00:53.017311 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:53.017289    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mdk\" (UniqueName: \"kubernetes.io/projected/0c5cf9bb-a540-4b52-82d6-d147fc112377-kube-api-access-n7mdk\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:53.017311 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:53.017310    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c5cf9bb-a540-4b52-82d6-d147fc112377-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:00:53.156669 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:53.156638    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"]
Oct 30 19:00:53.161918 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:53.161884    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-69pslkw"]
Oct 30 19:00:54.056093 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:00:54.056061    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" path="/var/lib/kubelet/pods/0c5cf9bb-a540-4b52-82d6-d147fc112377/volumes"
Oct 30 19:00:58.080942 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:00:58.080909    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:01:01.308733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.308701    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"]
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309066    5025 cpu_manager.go:414] "containerMap: removing container" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="main"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309076    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="main"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309095    5025 cpu_manager.go:414] "containerMap: removing container" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="llm-d-routing-sidecar"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309100    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="llm-d-routing-sidecar"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309110    5025 cpu_manager.go:414] "containerMap: removing container" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309116    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309194    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="main"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309202    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8c7df5d-4794-4930-9e1b-28499ccce993" containerName="llm-d-routing-sidecar"
Oct 30 19:01:01.309268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.309210    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5cf9bb-a540-4b52-82d6-d147fc112377" containerName="main"
Oct 30 19:01:01.313721 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.313703    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.398639 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.398613    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Oct 30 19:01:01.398639 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.398629    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-wtv72\""
Oct 30 19:01:01.398838 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.398802    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Oct 30 19:01:01.398838 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.398812    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:01:01.417961 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.417938    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"]
Oct 30 19:01:01.497002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.496963    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-credential-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497004    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-istio-token\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497038    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-data\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497126    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-certs\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497160    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mztmg\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-kube-api-access-mztmg\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497307 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497184    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497307 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497255    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-envoy\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497400 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497298    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dee6562-7392-4385-8105-92cd13e32ebd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.497400 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.497394    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dee6562-7392-4385-8105-92cd13e32ebd-istio-podinfo\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598196 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598112    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dee6562-7392-4385-8105-92cd13e32ebd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598196 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598174    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dee6562-7392-4385-8105-92cd13e32ebd-istio-podinfo\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598208    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-credential-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598233    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-istio-token\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598255    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-data\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598299    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-certs\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598369    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mztmg\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-kube-api-access-mztmg\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598405    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598439    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-envoy\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598942 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598912    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.598942 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598932    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-data\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.599127 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.598982    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-certs\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.599127 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.599011    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-credential-socket\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.599127 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.599058    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dee6562-7392-4385-8105-92cd13e32ebd-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.600826 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.600807    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-envoy\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.600826 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.600818    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dee6562-7392-4385-8105-92cd13e32ebd-istio-podinfo\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.654141 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.654113    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-istio-token\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.654506 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.654480    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mztmg\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-kube-api-access-mztmg\") pod \"router-gateway-1-openshift-default-dcc9f996f-9zl6x\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:01.926746 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:01.926700    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:02.408065 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:02.408034    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"]
Oct 30 19:01:02.447602 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:01:02.447574    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dee6562_7392_4385_8105_92cd13e32ebd.slice/crio-74a2b6864fc37b7f7c6f03e25cb1f3b7472d6fa2a27f69bb9206ce2ecb09d8c1 WatchSource:0}: Error finding container 74a2b6864fc37b7f7c6f03e25cb1f3b7472d6fa2a27f69bb9206ce2ecb09d8c1: Status 404 returned error can't find the container with id 74a2b6864fc37b7f7c6f03e25cb1f3b7472d6fa2a27f69bb9206ce2ecb09d8c1
Oct 30 19:01:02.449625 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:02.449589    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:02.449723 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:02.449671    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:02.449791 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:02.449718    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:01:03.046001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:03.045961    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x" event={"ID":"3dee6562-7392-4385-8105-92cd13e32ebd","Type":"ContainerStarted","Data":"08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb"}
Oct 30 19:01:03.046187 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:03.046022    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x" event={"ID":"3dee6562-7392-4385-8105-92cd13e32ebd","Type":"ContainerStarted","Data":"74a2b6864fc37b7f7c6f03e25cb1f3b7472d6fa2a27f69bb9206ce2ecb09d8c1"}
Oct 30 19:01:03.166550 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:03.166503    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x" podStartSLOduration=2.16648956 podStartE2EDuration="2.16648956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:03.164750068 +0000 UTC m=+1445.245336275" watchObservedRunningTime="2026-10-30 19:01:03.16648956 +0000 UTC m=+1445.247075820"
Oct 30 19:01:03.927699 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:03.927664    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:03.932720 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:03.932694    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:04.055975 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:04.055944    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:04.056134 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:04.056031    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:08.325384 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.325351    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"]
Oct 30 19:01:08.329113 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.329091    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:08.357423 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.357393    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20c7d50-3829-46e9-aa8f-db0215130b08-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schednbqs9\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:08.357566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.357483    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqhg2\" (UniqueName: \"kubernetes.io/projected/d20c7d50-3829-46e9-aa8f-db0215130b08-kube-api-access-jqhg2\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schednbqs9\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:08.421220 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.421189    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-9w5px\""
Oct 30 19:01:08.430915 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.430882    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Oct 30 19:01:08.431077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.430951    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Oct 30 19:01:08.431077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.430961    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Oct 30 19:01:08.431077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.430999    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-nq5zm\""
Oct 30 19:01:08.458443 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.458416    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20c7d50-3829-46e9-aa8f-db0215130b08-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schednbqs9\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:08.458589 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.458462    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqhg2\" (UniqueName: \"kubernetes.io/projected/d20c7d50-3829-46e9-aa8f-db0215130b08-kube-api-access-jqhg2\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schednbqs9\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:08.460870 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.460846    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20c7d50-3829-46e9-aa8f-db0215130b08-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schednbqs9\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:08.470599 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.470577    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"]
Oct 30 19:01:08.521766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.521732    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqhg2\" (UniqueName: \"kubernetes.io/projected/d20c7d50-3829-46e9-aa8f-db0215130b08-kube-api-access-jqhg2\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schednbqs9\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:08.643447 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:08.643399    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:09.012929 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:09.012855    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"]
Oct 30 19:01:09.018836 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:01:09.018810    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20c7d50_3829_46e9_aa8f_db0215130b08.slice/crio-b704bb9ff2984e84d8cd940b6c5f1cd13f94aa17618ae1d62a7c25447c976993 WatchSource:0}: Error finding container b704bb9ff2984e84d8cd940b6c5f1cd13f94aa17618ae1d62a7c25447c976993: Status 404 returned error can't find the container with id b704bb9ff2984e84d8cd940b6c5f1cd13f94aa17618ae1d62a7c25447c976993
Oct 30 19:01:09.072784 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:09.072756    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" event={"ID":"d20c7d50-3829-46e9-aa8f-db0215130b08","Type":"ContainerStarted","Data":"b704bb9ff2984e84d8cd940b6c5f1cd13f94aa17618ae1d62a7c25447c976993"}
Oct 30 19:01:10.078599 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:10.078565    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" event={"ID":"d20c7d50-3829-46e9-aa8f-db0215130b08","Type":"ContainerStarted","Data":"55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818"}
Oct 30 19:01:10.079002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:10.078658    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:10.188519 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:10.188471    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" podStartSLOduration=2.18845333 podStartE2EDuration="2.18845333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:10.187510808 +0000 UTC m=+1452.268097036" watchObservedRunningTime="2026-10-30 19:01:10.18845333 +0000 UTC m=+1452.269039536"
Oct 30 19:01:20.305992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:20.305957    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"]
Oct 30 19:01:20.306454 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:20.306141    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x" podUID="3dee6562-7392-4385-8105-92cd13e32ebd" containerName="istio-proxy" containerID="cri-o://08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb" gracePeriod=30
Oct 30 19:01:25.564150 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.564092    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:25.697894 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.697858    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dee6562-7392-4385-8105-92cd13e32ebd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dee6562-7392-4385-8105-92cd13e32ebd-istio-podinfo\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.697894 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.697895    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-istio-token\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-istio-token\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698104 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.697916    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-data\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698104 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.697936    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-socket\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698104 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.697954    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-certs\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698104 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698039    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-envoy\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698102    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dee6562-7392-4385-8105-92cd13e32ebd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dee6562-7392-4385-8105-92cd13e32ebd-istiod-ca-cert\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698149    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-credential-socket\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698200    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-kube-api-access-mztmg\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-kube-api-access-mztmg\") pod \"3dee6562-7392-4385-8105-92cd13e32ebd\" (UID: \"3dee6562-7392-4385-8105-92cd13e32ebd\") "
Oct 30 19:01:25.698287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698244    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-data" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:25.698287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698263    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-socket" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:25.698511 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698382    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-certs" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:25.698569 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698521    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.698569 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698543    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.698569 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698557    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.698721 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698624    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-credential-socket" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:25.698774 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.698721    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dee6562-7392-4385-8105-92cd13e32ebd-istiod-ca-cert" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:01:25.700300 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.700274    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-istio-token" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:25.700453 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.700425    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-envoy" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:01:25.700544 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.700526    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dee6562-7392-4385-8105-92cd13e32ebd-istio-podinfo" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:01:25.700698 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.700676    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-kube-api-access-mztmg" pod "3dee6562-7392-4385-8105-92cd13e32ebd" (UID: "3dee6562-7392-4385-8105-92cd13e32ebd"). InnerVolumeSpecName "kube-api-access-mztmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:01:25.799287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.799252    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.799287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.799290    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dee6562-7392-4385-8105-92cd13e32ebd-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.799487 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.799307    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dee6562-7392-4385-8105-92cd13e32ebd-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.799487 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.799338    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mztmg\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-kube-api-access-mztmg\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.799487 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.799356    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dee6562-7392-4385-8105-92cd13e32ebd-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:25.799487 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:25.799370    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dee6562-7392-4385-8105-92cd13e32ebd-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:01:26.219632 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.219597    5025 generic.go:356] "Generic (PLEG): container finished" podID="3dee6562-7392-4385-8105-92cd13e32ebd" containerID="08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb" exitCode=0
Oct 30 19:01:26.219818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.219649    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"
Oct 30 19:01:26.219818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.219686    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x" event={"ID":"3dee6562-7392-4385-8105-92cd13e32ebd","Type":"ContainerDied","Data":"08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb"}
Oct 30 19:01:26.219818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.219744    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x" event={"ID":"3dee6562-7392-4385-8105-92cd13e32ebd","Type":"ContainerDied","Data":"74a2b6864fc37b7f7c6f03e25cb1f3b7472d6fa2a27f69bb9206ce2ecb09d8c1"}
Oct 30 19:01:26.219818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.219779    5025 scope.go:122] "RemoveContainer" containerID="08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb"
Oct 30 19:01:26.228477 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.228458    5025 scope.go:122] "RemoveContainer" containerID="08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb"
Oct 30 19:01:26.228740 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:01:26.228720    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb\": container with ID starting with 08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb not found: ID does not exist" containerID="08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb"
Oct 30 19:01:26.228800 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.228754    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb"} err="failed to get container status \"08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb\": rpc error: code = NotFound desc = could not find container \"08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb\": container with ID starting with 08d8d06a012752c68631387d228c25b9367c91e4baba8988ba895b02ebb8d3bb not found: ID does not exist"
Oct 30 19:01:26.379529 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.379498    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"]
Oct 30 19:01:26.427044 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:26.427015    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-dcc9f996f-9zl6x"]
Oct 30 19:01:28.056506 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:28.056463    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dee6562-7392-4385-8105-92cd13e32ebd" path="/var/lib/kubelet/pods/3dee6562-7392-4385-8105-92cd13e32ebd/volumes"
Oct 30 19:01:28.081959 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:01:28.081929    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:01:41.086005 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:41.085932    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:01:49.454921 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.454886    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"]
Oct 30 19:01:49.455304 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.455287    5025 cpu_manager.go:414] "containerMap: removing container" podUID="3dee6562-7392-4385-8105-92cd13e32ebd" containerName="istio-proxy"
Oct 30 19:01:49.455304 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.455299    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dee6562-7392-4385-8105-92cd13e32ebd" containerName="istio-proxy"
Oct 30 19:01:49.455417 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.455390    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dee6562-7392-4385-8105-92cd13e32ebd" containerName="istio-proxy"
Oct 30 19:01:49.458552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.458531    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.504526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.504496    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"]
Oct 30 19:01:49.549835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.549805    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Oct 30 19:01:49.550049 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.550022    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Oct 30 19:01:49.550162 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.550056    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:01:49.550162 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.550093    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Oct 30 19:01:49.550162 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.550132    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-fwgqr\""
Oct 30 19:01:49.550286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.550182    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Oct 30 19:01:49.617953 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.617917    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-kserve-provision-location\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.618124 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.617976    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69bed66d-c986-45f1-b166-4cbf4cd3f236-cabundle-cert\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.618124 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.618077    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-home\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.618124 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.618105    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-tmp-dir\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.618124 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.618122    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7vnq\" (UniqueName: \"kubernetes.io/projected/69bed66d-c986-45f1-b166-4cbf4cd3f236-kube-api-access-r7vnq\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.618346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.618197    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-dshm\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.618346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.618229    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69bed66d-c986-45f1-b166-4cbf4cd3f236-tls-certs\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.618346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.618267    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-model-cache\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719521 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719441    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-kserve-provision-location\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719521 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719485    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69bed66d-c986-45f1-b166-4cbf4cd3f236-cabundle-cert\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719521 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719512    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-home\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719529    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-tmp-dir\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719544    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7vnq\" (UniqueName: \"kubernetes.io/projected/69bed66d-c986-45f1-b166-4cbf4cd3f236-kube-api-access-r7vnq\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719565    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-dshm\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719589    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69bed66d-c986-45f1-b166-4cbf4cd3f236-tls-certs\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.719815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719630    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-model-cache\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.720053 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.719958    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-home\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.720053 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.720005    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-kserve-provision-location\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.720154 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.720084    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-tmp-dir\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.720154 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.720114    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-model-cache\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.720291 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.720272    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69bed66d-c986-45f1-b166-4cbf4cd3f236-cabundle-cert\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.722015 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.721990    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-dshm\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.722263 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.722247    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69bed66d-c986-45f1-b166-4cbf4cd3f236-tls-certs\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:49.778257 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:49.778236    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7vnq\" (UniqueName: \"kubernetes.io/projected/69bed66d-c986-45f1-b166-4cbf4cd3f236-kube-api-access-r7vnq\") pod \"auth-enabled-test-kserve-58d9587c87-hsl9m\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:50.075144 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:50.075060    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:50.552876 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:50.552832    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"]
Oct 30 19:01:50.557715 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:01:50.557683    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bed66d_c986_45f1_b166_4cbf4cd3f236.slice/crio-1461d01cc7fb15dcdf610db8a15e90396595521969cecd15df37c86b5b1bc08e WatchSource:0}: Error finding container 1461d01cc7fb15dcdf610db8a15e90396595521969cecd15df37c86b5b1bc08e: Status 404 returned error can't find the container with id 1461d01cc7fb15dcdf610db8a15e90396595521969cecd15df37c86b5b1bc08e
Oct 30 19:01:51.338418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:51.338380    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" event={"ID":"69bed66d-c986-45f1-b166-4cbf4cd3f236","Type":"ContainerStarted","Data":"fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227"}
Oct 30 19:01:51.338715 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:51.338443    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" event={"ID":"69bed66d-c986-45f1-b166-4cbf4cd3f236","Type":"ContainerStarted","Data":"1461d01cc7fb15dcdf610db8a15e90396595521969cecd15df37c86b5b1bc08e"}
Oct 30 19:01:53.349823 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:53.349785    5025 generic.go:356] "Generic (PLEG): container finished" podID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerID="fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227" exitCode=0
Oct 30 19:01:53.350234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:53.349865    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" event={"ID":"69bed66d-c986-45f1-b166-4cbf4cd3f236","Type":"ContainerDied","Data":"fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227"}
Oct 30 19:01:54.358482 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:54.358396    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" event={"ID":"69bed66d-c986-45f1-b166-4cbf4cd3f236","Type":"ContainerStarted","Data":"a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9"}
Oct 30 19:01:54.569879 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:54.569830    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podStartSLOduration=3.377813841 podStartE2EDuration="5.569816223s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.192002382s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:54.521213379 +0000 UTC m=+1496.601799585" watchObservedRunningTime="2026-10-30 19:01:54.569816223 +0000 UTC m=+1496.650402430"
Oct 30 19:01:55.075533 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:55.075459    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:01:56.108305 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:56.108267    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"]
Oct 30 19:01:58.083582 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:01:58.083541    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:01:58.285219 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:58.285191    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:01:58.286303 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:01:58.286275    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:02:00.076286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:00.076255    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:02:00.077998 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:00.077969    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:02:01.086235 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:01.086196    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:02:10.076142 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:10.076094    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:02:11.085275 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:11.085230    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:02:11.129911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:11.129872    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main" containerID="cri-o://55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818" gracePeriod=45
Oct 30 19:02:20.076418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:20.076375    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:02:21.085428 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:21.085387    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:02:21.085792 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:21.085482    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:02:28.082354 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:02:28.082296    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:02:30.075858 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:30.075812    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:02:31.085084 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:31.085046    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:02:40.075605 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:40.075555    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:02:41.084882 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.084847    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:02:41.393856 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.393827    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:02:41.506651 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.506618    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d20c7d50-3829-46e9-aa8f-db0215130b08-tls-certs\" (UniqueName: \"kubernetes.io/secret/d20c7d50-3829-46e9-aa8f-db0215130b08-tls-certs\") pod \"d20c7d50-3829-46e9-aa8f-db0215130b08\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") "
Oct 30 19:02:41.506840 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.506661    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20c7d50-3829-46e9-aa8f-db0215130b08-kube-api-access-jqhg2\" (UniqueName: \"kubernetes.io/projected/d20c7d50-3829-46e9-aa8f-db0215130b08-kube-api-access-jqhg2\") pod \"d20c7d50-3829-46e9-aa8f-db0215130b08\" (UID: \"d20c7d50-3829-46e9-aa8f-db0215130b08\") "
Oct 30 19:02:41.508860 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.508831    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20c7d50-3829-46e9-aa8f-db0215130b08-tls-certs" pod "d20c7d50-3829-46e9-aa8f-db0215130b08" (UID: "d20c7d50-3829-46e9-aa8f-db0215130b08"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:02:41.509616 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.509144    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20c7d50-3829-46e9-aa8f-db0215130b08-kube-api-access-jqhg2" pod "d20c7d50-3829-46e9-aa8f-db0215130b08" (UID: "d20c7d50-3829-46e9-aa8f-db0215130b08"). InnerVolumeSpecName "kube-api-access-jqhg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:02:41.602524 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.602493    5025 generic.go:356] "Generic (PLEG): container finished" podID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerID="55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818" exitCode=0
Oct 30 19:02:41.602682 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.602537    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" event={"ID":"d20c7d50-3829-46e9-aa8f-db0215130b08","Type":"ContainerDied","Data":"55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818"}
Oct 30 19:02:41.602682 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.602542    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"
Oct 30 19:02:41.602682 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.602577    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9" event={"ID":"d20c7d50-3829-46e9-aa8f-db0215130b08","Type":"ContainerDied","Data":"b704bb9ff2984e84d8cd940b6c5f1cd13f94aa17618ae1d62a7c25447c976993"}
Oct 30 19:02:41.602682 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.602598    5025 scope.go:122] "RemoveContainer" containerID="55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818"
Oct 30 19:02:41.607676 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.607648    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20c7d50-3829-46e9-aa8f-db0215130b08-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:41.607676 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.607675    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqhg2\" (UniqueName: \"kubernetes.io/projected/d20c7d50-3829-46e9-aa8f-db0215130b08-kube-api-access-jqhg2\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:02:41.612410 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.612391    5025 scope.go:122] "RemoveContainer" containerID="55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818"
Oct 30 19:02:41.612683 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:02:41.612663    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818\": container with ID starting with 55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818 not found: ID does not exist" containerID="55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818"
Oct 30 19:02:41.612746 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.612696    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818"} err="failed to get container status \"55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818\": rpc error: code = NotFound desc = could not find container \"55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818\": container with ID starting with 55b25cad9686eeb26ce422bbb125ff07c9d04e25b6c13641d2f9afe126b09818 not found: ID does not exist"
Oct 30 19:02:41.768134 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.768039    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"]
Oct 30 19:02:41.790340 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:41.790294    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schednbqs9"]
Oct 30 19:02:42.058044 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:42.057966    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" path="/var/lib/kubelet/pods/d20c7d50-3829-46e9-aa8f-db0215130b08/volumes"
Oct 30 19:02:50.075956 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:50.075908    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:02:51.686689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.686654    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"]
Oct 30 19:02:51.687284 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.687262    5025 cpu_manager.go:414] "containerMap: removing container" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main"
Oct 30 19:02:51.687396 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.687286    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main"
Oct 30 19:02:51.687465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.687408    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20c7d50-3829-46e9-aa8f-db0215130b08" containerName="main"
Oct 30 19:02:51.690867 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.690844    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.747358 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.747308    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:02:51.747358 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.747308    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Oct 30 19:02:51.747358 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.747354    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Oct 30 19:02:51.747615 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.747416    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Oct 30 19:02:51.794763 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.794733    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-j2j9z\""
Oct 30 19:02:51.794763 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.794729    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Oct 30 19:02:51.797223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.797199    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"]
Oct 30 19:02:51.810596 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.810567    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.810719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.810700    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.810763 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.810729    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-home\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.810763 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.810756    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.810827 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.810790    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-dshm\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.810988 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.810964    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b626f597-3ae8-4683-9b3a-ae84badb1f0b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.811047 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.811018    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-model-cache\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.811084 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.811063    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mp6t\" (UniqueName: \"kubernetes.io/projected/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kube-api-access-5mp6t\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911581 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911543    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-model-cache\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911581 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911585    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mp6t\" (UniqueName: \"kubernetes.io/projected/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kube-api-access-5mp6t\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911845 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911620    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911845 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911679    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911845 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911699    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-home\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911845 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911723    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911845 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911762    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-dshm\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.911845 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.911830    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b626f597-3ae8-4683-9b3a-ae84badb1f0b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.912163 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.912085    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-model-cache\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.912230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.912181    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-home\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.912230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.912213    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.912490 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.912468    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.912690 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.912670    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b626f597-3ae8-4683-9b3a-ae84badb1f0b-cabundle-cert\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.914285 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.914264    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.914510 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.914489    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-dshm\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:51.967876 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:51.967793    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mp6t\" (UniqueName: \"kubernetes.io/projected/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kube-api-access-5mp6t\") pod \"tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:52.009340 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.009290    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:52.330513 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.330428    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"]
Oct 30 19:02:52.335177 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.335155    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.395693 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.395658    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"]
Oct 30 19:02:52.416114 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.416086    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089b6297-a74e-4853-9888-ad7627d64546-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.416274 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.416137    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.416274 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.416253    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.416395 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.416304    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089b6297-a74e-4853-9888-ad7627d64546-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.438953 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.438914    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"]
Oct 30 19:02:52.443584 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:02:52.443556    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb626f597_3ae8_4683_9b3a_ae84badb1f0b.slice/crio-225a9ad0363351cc6e63d4c5ae26e91f5041212e479efb423555d8da03026552 WatchSource:0}: Error finding container 225a9ad0363351cc6e63d4c5ae26e91f5041212e479efb423555d8da03026552: Status 404 returned error can't find the container with id 225a9ad0363351cc6e63d4c5ae26e91f5041212e479efb423555d8da03026552
Oct 30 19:02:52.517373 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.517313    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089b6297-a74e-4853-9888-ad7627d64546-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.517502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.517389    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.517502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.517442    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.517502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.517472    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089b6297-a74e-4853-9888-ad7627d64546-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.518314 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.518292    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089b6297-a74e-4853-9888-ad7627d64546-cabundle-cert\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.519687 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.518993    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.519687 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.519051    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.521114 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.521090    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089b6297-a74e-4853-9888-ad7627d64546-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.655263 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.655226    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:52.659863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.659824    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" event={"ID":"b626f597-3ae8-4683-9b3a-ae84badb1f0b","Type":"ContainerStarted","Data":"451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f"}
Oct 30 19:02:52.660016 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:52.659904    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" event={"ID":"b626f597-3ae8-4683-9b3a-ae84badb1f0b","Type":"ContainerStarted","Data":"225a9ad0363351cc6e63d4c5ae26e91f5041212e479efb423555d8da03026552"}
Oct 30 19:02:53.038766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:53.038671    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"]
Oct 30 19:02:53.044285 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:02:53.044252    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089b6297_a74e_4853_9888_ad7627d64546.slice/crio-9ed6df6d4cfb0360e57d013a7a248827b4cac291dfcf6e3e125d68bb49f0e3c2 WatchSource:0}: Error finding container 9ed6df6d4cfb0360e57d013a7a248827b4cac291dfcf6e3e125d68bb49f0e3c2: Status 404 returned error can't find the container with id 9ed6df6d4cfb0360e57d013a7a248827b4cac291dfcf6e3e125d68bb49f0e3c2
Oct 30 19:02:53.671788 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:53.671741    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" event={"ID":"089b6297-a74e-4853-9888-ad7627d64546","Type":"ContainerStarted","Data":"4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0"}
Oct 30 19:02:53.672010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:53.671822    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" event={"ID":"089b6297-a74e-4853-9888-ad7627d64546","Type":"ContainerStarted","Data":"9ed6df6d4cfb0360e57d013a7a248827b4cac291dfcf6e3e125d68bb49f0e3c2"}
Oct 30 19:02:55.686151 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:55.686119    5025 generic.go:356] "Generic (PLEG): container finished" podID="089b6297-a74e-4853-9888-ad7627d64546" containerID="4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0" exitCode=0
Oct 30 19:02:55.686574 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:55.686196    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" event={"ID":"089b6297-a74e-4853-9888-ad7627d64546","Type":"ContainerDied","Data":"4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0"}
Oct 30 19:02:55.687860 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:55.687836    5025 generic.go:356] "Generic (PLEG): container finished" podID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerID="451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f" exitCode=0
Oct 30 19:02:55.687977 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:55.687866    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" event={"ID":"b626f597-3ae8-4683-9b3a-ae84badb1f0b","Type":"ContainerDied","Data":"451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f"}
Oct 30 19:02:56.695259 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:56.695214    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" event={"ID":"089b6297-a74e-4853-9888-ad7627d64546","Type":"ContainerStarted","Data":"ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7"}
Oct 30 19:02:56.695779 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:56.695615    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:02:56.697445 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:56.697419    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" event={"ID":"b626f597-3ae8-4683-9b3a-ae84badb1f0b","Type":"ContainerStarted","Data":"a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85"}
Oct 30 19:02:56.808110 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:56.808046    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" podStartSLOduration=2.316464915 podStartE2EDuration="4.808027276s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.491562361s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:56.806898763 +0000 UTC m=+1558.887484971" watchObservedRunningTime="2026-10-30 19:02:56.808027276 +0000 UTC m=+1558.888613485"
Oct 30 19:02:56.921446 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:56.921385    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" podStartSLOduration=3.3631136169999998 podStartE2EDuration="5.921367091s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.558253474s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:02:56.920047423 +0000 UTC m=+1559.000633690" watchObservedRunningTime="2026-10-30 19:02:56.921367091 +0000 UTC m=+1559.001953299"
Oct 30 19:02:57.009640 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:02:57.009541    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:02:58.083824 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:02:58.083787    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:03:00.076505 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:00.076464    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:03:02.010443 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:02.010399    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:03:02.023502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:02.023468    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:03:02.741002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:02.740969    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:03:10.076081 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:10.075985    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:03:20.076113 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:20.076058    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:03:27.714427 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:27.714391    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:03:28.082668 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:28.082585    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:03:30.076313 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:30.076269    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:03:32.711457 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:32.711423    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh_089b6297-a74e-4853-9888-ad7627d64546/main/0.log"
Oct 30 19:03:35.956512 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:35.956477    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"]
Oct 30 19:03:35.956924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:35.956755    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" podUID="089b6297-a74e-4853-9888-ad7627d64546" containerName="main" containerID="cri-o://ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7" gracePeriod=30
Oct 30 19:03:36.000614 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.000581    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"]
Oct 30 19:03:36.010440 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:36.010395    5025 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85" cmd=["/bin/sleep","15"]
Oct 30 19:03:36.010584 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.010463    5025 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:03:36.010584 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:36.010484    5025 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="main" containerID="cri-o://a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85"
Oct 30 19:03:36.010584 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.010520    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="main" containerID="cri-o://a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85" gracePeriod=60
Oct 30 19:03:36.628742 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.628685    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:03:36.758747 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.758712    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:03:36.769081 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.769049    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-kserve-provision-location\") pod \"089b6297-a74e-4853-9888-ad7627d64546\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") "
Oct 30 19:03:36.769234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.769148    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/089b6297-a74e-4853-9888-ad7627d64546-tls-certs\" (UniqueName: \"kubernetes.io/secret/089b6297-a74e-4853-9888-ad7627d64546-tls-certs\") pod \"089b6297-a74e-4853-9888-ad7627d64546\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") "
Oct 30 19:03:36.769303 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.769274    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-tmp-dir\") pod \"089b6297-a74e-4853-9888-ad7627d64546\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") "
Oct 30 19:03:36.769379 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.769305    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089b6297-a74e-4853-9888-ad7627d64546-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089b6297-a74e-4853-9888-ad7627d64546-cabundle-cert\") pod \"089b6297-a74e-4853-9888-ad7627d64546\" (UID: \"089b6297-a74e-4853-9888-ad7627d64546\") "
Oct 30 19:03:36.769758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.769728    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089b6297-a74e-4853-9888-ad7627d64546-cabundle-cert" pod "089b6297-a74e-4853-9888-ad7627d64546" (UID: "089b6297-a74e-4853-9888-ad7627d64546"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:36.769865 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.769731    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-tmp-dir" pod "089b6297-a74e-4853-9888-ad7627d64546" (UID: "089b6297-a74e-4853-9888-ad7627d64546"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.771524 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.771495    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089b6297-a74e-4853-9888-ad7627d64546-tls-certs" pod "089b6297-a74e-4853-9888-ad7627d64546" (UID: "089b6297-a74e-4853-9888-ad7627d64546"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:36.790317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.790280    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-kserve-provision-location" pod "089b6297-a74e-4853-9888-ad7627d64546" (UID: "089b6297-a74e-4853-9888-ad7627d64546"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.870102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870068    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kube-api-access-5mp6t\" (UniqueName: \"kubernetes.io/projected/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kube-api-access-5mp6t\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870131    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-dshm\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870170    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b626f597-3ae8-4683-9b3a-ae84badb1f0b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b626f597-3ae8-4683-9b3a-ae84badb1f0b-cabundle-cert\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870206    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kserve-provision-location\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870256    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-home\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-home\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870288    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-model-cache\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870607 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870353    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tls-certs\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870607 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870416    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tmp-dir\") pod \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\" (UID: \"b626f597-3ae8-4683-9b3a-ae84badb1f0b\") "
Oct 30 19:03:36.870718 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870620    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-home" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.870718 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870691    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-model-cache" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.870810 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870716    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b626f597-3ae8-4683-9b3a-ae84badb1f0b-cabundle-cert" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:03:36.870859 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870833    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.870859 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870846    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tmp-dir" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.870965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870855    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/089b6297-a74e-4853-9888-ad7627d64546-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.870965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870873    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/089b6297-a74e-4853-9888-ad7627d64546-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.870965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870888    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b626f597-3ae8-4683-9b3a-ae84badb1f0b-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.870965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870901    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089b6297-a74e-4853-9888-ad7627d64546-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.870965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870913    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.870965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.870921    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.872874 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.872847    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kube-api-access-5mp6t" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "kube-api-access-5mp6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:03:36.872979 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.872940    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tls-certs" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:03:36.873100 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.873077    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-dshm" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.893557 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.893475    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kserve-provision-location" pod "b626f597-3ae8-4683-9b3a-ae84badb1f0b" (UID: "b626f597-3ae8-4683-9b3a-ae84badb1f0b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:03:36.924905 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.924872    5025 generic.go:356] "Generic (PLEG): container finished" podID="089b6297-a74e-4853-9888-ad7627d64546" containerID="ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7" exitCode=0
Oct 30 19:03:36.925091 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.924968    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"
Oct 30 19:03:36.925091 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.924961    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" event={"ID":"089b6297-a74e-4853-9888-ad7627d64546","Type":"ContainerDied","Data":"ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7"}
Oct 30 19:03:36.925234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.925218    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh" event={"ID":"089b6297-a74e-4853-9888-ad7627d64546","Type":"ContainerDied","Data":"9ed6df6d4cfb0360e57d013a7a248827b4cac291dfcf6e3e125d68bb49f0e3c2"}
Oct 30 19:03:36.925296 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.925256    5025 scope.go:122] "RemoveContainer" containerID="ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7"
Oct 30 19:03:36.927166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.927146    5025 generic.go:356] "Generic (PLEG): container finished" podID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerID="a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85" exitCode=0
Oct 30 19:03:36.927269 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.927199    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" event={"ID":"b626f597-3ae8-4683-9b3a-ae84badb1f0b","Type":"ContainerDied","Data":"a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85"}
Oct 30 19:03:36.927269 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.927220    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"
Oct 30 19:03:36.927269 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.927256    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" event={"ID":"b626f597-3ae8-4683-9b3a-ae84badb1f0b","Type":"ContainerDied","Data":"225a9ad0363351cc6e63d4c5ae26e91f5041212e479efb423555d8da03026552"}
Oct 30 19:03:36.934842 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.934820    5025 scope.go:122] "RemoveContainer" containerID="4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0"
Oct 30 19:03:36.944676 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.944652    5025 scope.go:122] "RemoveContainer" containerID="ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7"
Oct 30 19:03:36.944939 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:36.944921    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7\": container with ID starting with ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7 not found: ID does not exist" containerID="ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7"
Oct 30 19:03:36.945003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.944948    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7"} err="failed to get container status \"ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7\": rpc error: code = NotFound desc = could not find container \"ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7\": container with ID starting with ade63ebdb60d805246dffddf6393a7a89380b6c719c012ee2fda3248de0926d7 not found: ID does not exist"
Oct 30 19:03:36.945003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.944965    5025 scope.go:122] "RemoveContainer" containerID="4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0"
Oct 30 19:03:36.945164 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:36.945150    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0\": container with ID starting with 4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0 not found: ID does not exist" containerID="4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0"
Oct 30 19:03:36.945205 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.945168    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0"} err="failed to get container status \"4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0\": rpc error: code = NotFound desc = could not find container \"4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0\": container with ID starting with 4d46ab2ee4ec4bf07c7450f2f54df2a65995859687a14959d06fd9eed6dde8d0 not found: ID does not exist"
Oct 30 19:03:36.945205 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.945179    5025 scope.go:122] "RemoveContainer" containerID="a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85"
Oct 30 19:03:36.953806 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.953788    5025 scope.go:122] "RemoveContainer" containerID="451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f"
Oct 30 19:03:36.961689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.961548    5025 scope.go:122] "RemoveContainer" containerID="a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85"
Oct 30 19:03:36.961930 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:36.961824    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85\": container with ID starting with a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85 not found: ID does not exist" containerID="a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85"
Oct 30 19:03:36.961930 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.961860    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85"} err="failed to get container status \"a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85\": rpc error: code = NotFound desc = could not find container \"a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85\": container with ID starting with a2b1a00d5d9f92d1ae5b3facd83cb5cd84a8091f6af17fe5b44ed9f61e7aaf85 not found: ID does not exist"
Oct 30 19:03:36.961930 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.961884    5025 scope.go:122] "RemoveContainer" containerID="451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f"
Oct 30 19:03:36.962135 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:36.962116    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f\": container with ID starting with 451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f not found: ID does not exist" containerID="451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f"
Oct 30 19:03:36.962180 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.962141    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f"} err="failed to get container status \"451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f\": rpc error: code = NotFound desc = could not find container \"451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f\": container with ID starting with 451e1991008b98e8ede4bea3b889873a151f1223fdde1f6fc770942bbb03921f not found: ID does not exist"
Oct 30 19:03:36.971913 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.971885    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mp6t\" (UniqueName: \"kubernetes.io/projected/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kube-api-access-5mp6t\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.971983 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.971921    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.971983 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.971935    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.971983 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.971951    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:36.971983 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:36.971964    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b626f597-3ae8-4683-9b3a-ae84badb1f0b-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:03:37.089102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:37.089069    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"]
Oct 30 19:03:37.104181 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:37.104143    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k"]
Oct 30 19:03:37.214680 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:37.214597    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"]
Oct 30 19:03:37.219092 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:37.219066    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-79c76f54ff-9mwbh"]
Oct 30 19:03:37.730391 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:37.730350    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74875f6fcb-5lf7k" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:03:38.056606 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:38.056532    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089b6297-a74e-4853-9888-ad7627d64546" path="/var/lib/kubelet/pods/089b6297-a74e-4853-9888-ad7627d64546/volumes"
Oct 30 19:03:38.057077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:38.057062    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" path="/var/lib/kubelet/pods/b626f597-3ae8-4683-9b3a-ae84badb1f0b/volumes"
Oct 30 19:03:40.076045 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:40.076006    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": dial tcp 10.132.0.61:8000: connect: connection refused"
Oct 30 19:03:50.085960 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:50.085925    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:03:50.093527 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:03:50.093507    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:03:58.081108 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:03:58.081071    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:04:00.096542 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:00.096505    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"]
Oct 30 19:04:15.123595 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:15.123536    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main" containerID="cri-o://a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9" gracePeriod=45
Oct 30 19:04:15.864499 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:15.864476    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:04:16.043977 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.043895    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-tmp-dir\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.043977 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.043936    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-home\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-home\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.044206 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044000    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69bed66d-c986-45f1-b166-4cbf4cd3f236-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69bed66d-c986-45f1-b166-4cbf4cd3f236-cabundle-cert\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.044206 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044088    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-dshm\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.044206 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044131    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bed66d-c986-45f1-b166-4cbf4cd3f236-kube-api-access-r7vnq\" (UniqueName: \"kubernetes.io/projected/69bed66d-c986-45f1-b166-4cbf4cd3f236-kube-api-access-r7vnq\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.044206 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044171    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69bed66d-c986-45f1-b166-4cbf4cd3f236-tls-certs\" (UniqueName: \"kubernetes.io/secret/69bed66d-c986-45f1-b166-4cbf4cd3f236-tls-certs\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.044471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044218    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-kserve-provision-location\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.044471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044245    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-model-cache\") pod \"69bed66d-c986-45f1-b166-4cbf4cd3f236\" (UID: \"69bed66d-c986-45f1-b166-4cbf4cd3f236\") "
Oct 30 19:04:16.044636 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044498    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bed66d-c986-45f1-b166-4cbf4cd3f236-cabundle-cert" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:04:16.044803 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044779    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-home" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:16.044896 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.044871    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-model-cache" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:16.047009 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.046980    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69bed66d-c986-45f1-b166-4cbf4cd3f236-tls-certs" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:04:16.047108 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.047011    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-dshm" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:16.047108 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.047022    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bed66d-c986-45f1-b166-4cbf4cd3f236-kube-api-access-r7vnq" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "kube-api-access-r7vnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:04:16.056181 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.056160    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-tmp-dir" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:16.068532 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.068500    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-kserve-provision-location" pod "69bed66d-c986-45f1-b166-4cbf4cd3f236" (UID: "69bed66d-c986-45f1-b166-4cbf4cd3f236"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:04:16.120419 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.120385    5025 generic.go:356] "Generic (PLEG): container finished" podID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerID="a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9" exitCode=0
Oct 30 19:04:16.120557 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.120440    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"
Oct 30 19:04:16.120557 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.120440    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" event={"ID":"69bed66d-c986-45f1-b166-4cbf4cd3f236","Type":"ContainerDied","Data":"a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9"}
Oct 30 19:04:16.120557 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.120495    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m" event={"ID":"69bed66d-c986-45f1-b166-4cbf4cd3f236","Type":"ContainerDied","Data":"1461d01cc7fb15dcdf610db8a15e90396595521969cecd15df37c86b5b1bc08e"}
Oct 30 19:04:16.120557 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.120516    5025 scope.go:122] "RemoveContainer" containerID="a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9"
Oct 30 19:04:16.130522 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.130222    5025 scope.go:122] "RemoveContainer" containerID="fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227"
Oct 30 19:04:16.137719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.137700    5025 scope.go:122] "RemoveContainer" containerID="a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9"
Oct 30 19:04:16.137980 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:04:16.137960    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9\": container with ID starting with a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9 not found: ID does not exist" containerID="a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9"
Oct 30 19:04:16.138062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.137991    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9"} err="failed to get container status \"a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9\": rpc error: code = NotFound desc = could not find container \"a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9\": container with ID starting with a9f9563dcfa646bb1ece2881b311ee8b24f0c0e1214bdbeab3487383e14e53a9 not found: ID does not exist"
Oct 30 19:04:16.138062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.138015    5025 scope.go:122] "RemoveContainer" containerID="fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227"
Oct 30 19:04:16.138297 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:04:16.138280    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227\": container with ID starting with fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227 not found: ID does not exist" containerID="fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227"
Oct 30 19:04:16.138357 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.138304    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227"} err="failed to get container status \"fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227\": rpc error: code = NotFound desc = could not find container \"fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227\": container with ID starting with fae1d6daabfbce7f602c4964c16f45fb7028eb4db66803db20ffff7e8cfc9227 not found: ID does not exist"
Oct 30 19:04:16.145700 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145681    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69bed66d-c986-45f1-b166-4cbf4cd3f236-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.145758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145702    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.145758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145711    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7vnq\" (UniqueName: \"kubernetes.io/projected/69bed66d-c986-45f1-b166-4cbf4cd3f236-kube-api-access-r7vnq\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.145758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145722    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69bed66d-c986-45f1-b166-4cbf4cd3f236-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.145758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145731    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.145758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145739    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.145758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145747    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.145758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.145756    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69bed66d-c986-45f1-b166-4cbf4cd3f236-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:04:16.283942 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.283912    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"]
Oct 30 19:04:16.289444 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:16.289421    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-58d9587c87-hsl9m"]
Oct 30 19:04:18.055953 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:18.055924    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" path="/var/lib/kubelet/pods/69bed66d-c986-45f1-b166-4cbf4cd3f236/volumes"
Oct 30 19:04:28.081314 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:04:28.081270    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:04:34.206458 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.206415    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"]
Oct 30 19:04:34.207208 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207184    5025 cpu_manager.go:414] "containerMap: removing container" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="storage-initializer"
Oct 30 19:04:34.207297 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207213    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="storage-initializer"
Oct 30 19:04:34.207297 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207232    5025 cpu_manager.go:414] "containerMap: removing container" podUID="089b6297-a74e-4853-9888-ad7627d64546" containerName="storage-initializer"
Oct 30 19:04:34.207297 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207245    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089b6297-a74e-4853-9888-ad7627d64546" containerName="storage-initializer"
Oct 30 19:04:34.207297 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207256    5025 cpu_manager.go:414] "containerMap: removing container" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="main"
Oct 30 19:04:34.207297 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207264    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="main"
Oct 30 19:04:34.207564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207308    5025 cpu_manager.go:414] "containerMap: removing container" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main"
Oct 30 19:04:34.207564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207317    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main"
Oct 30 19:04:34.207564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207345    5025 cpu_manager.go:414] "containerMap: removing container" podUID="089b6297-a74e-4853-9888-ad7627d64546" containerName="main"
Oct 30 19:04:34.207564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207352    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089b6297-a74e-4853-9888-ad7627d64546" containerName="main"
Oct 30 19:04:34.207564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207412    5025 cpu_manager.go:414] "containerMap: removing container" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="storage-initializer"
Oct 30 19:04:34.207564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207420    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="storage-initializer"
Oct 30 19:04:34.207841 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207601    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b626f597-3ae8-4683-9b3a-ae84badb1f0b" containerName="main"
Oct 30 19:04:34.207841 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207622    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bed66d-c986-45f1-b166-4cbf4cd3f236" containerName="main"
Oct 30 19:04:34.207841 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.207636    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="089b6297-a74e-4853-9888-ad7627d64546" containerName="main"
Oct 30 19:04:34.212134 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.212112    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.265120 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.265081    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Oct 30 19:04:34.265120 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.265085    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Oct 30 19:04:34.290703 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.290670    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-model-cache\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.290891 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.290714    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.290891 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.290738    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-dshm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.290891 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.290798    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.291010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.290956    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6bgm\" (UniqueName: \"kubernetes.io/projected/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kube-api-access-p6bgm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.291010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.291001    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.291093 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.291036    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-home\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.291093 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.291080    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.312894 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.312696    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-kbdpg\""
Oct 30 19:04:34.312894 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.312830    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:04:34.313126 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.312921    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Oct 30 19:04:34.313126 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.312696    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Oct 30 19:04:34.315026 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.315004    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"]
Oct 30 19:04:34.391760 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.391728    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-home\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.391760 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.391762    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.391992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.391811    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-model-cache\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.391992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.391830    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.391992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.391848    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-dshm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.391992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.391968    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.392199 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.392046    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6bgm\" (UniqueName: \"kubernetes.io/projected/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kube-api-access-p6bgm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.392199 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.392100    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.392307 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.392213    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-home\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.392307 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.392234    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-model-cache\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.392480 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.392369    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.392538 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.392509    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.392630 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.392613    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-cabundle-cert\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.394257 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.394240    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-dshm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.394360 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.394346    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tls-certs\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.448251 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.448214    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6bgm\" (UniqueName: \"kubernetes.io/projected/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kube-api-access-p6bgm\") pod \"tokenizer-migration-path-test-kserve-75768b77f-w7bhk\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.527941 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.527850    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:34.839521 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.839434    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"]
Oct 30 19:04:34.844595 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.844571    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.895678 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.895648    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b292c2c3-c427-4e89-8936-7b011a88a1f0-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.895868 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.895691    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.895868 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.895837    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b292c2c3-c427-4e89-8936-7b011a88a1f0-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.895978 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.895874    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.904477 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.904452    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"]
Oct 30 19:04:34.948822 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.948794    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"]
Oct 30 19:04:34.961380 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:04:34.961348    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceba4cf6_f0e4_4a74_9a40_f1838bd4c6e0.slice/crio-a91d1970798722c0ee859ea2eb083962a0492a52d71b192a811ecef2432bed84 WatchSource:0}: Error finding container a91d1970798722c0ee859ea2eb083962a0492a52d71b192a811ecef2432bed84: Status 404 returned error can't find the container with id a91d1970798722c0ee859ea2eb083962a0492a52d71b192a811ecef2432bed84
Oct 30 19:04:34.996523 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.996492    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b292c2c3-c427-4e89-8936-7b011a88a1f0-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.996684 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.996534    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.996684 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.996662    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b292c2c3-c427-4e89-8936-7b011a88a1f0-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.996810 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.996699    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.997095 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.997066    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.997234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.997137    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.997712 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.997689    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b292c2c3-c427-4e89-8936-7b011a88a1f0-cabundle-cert\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:34.999055 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:34.999032    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b292c2c3-c427-4e89-8936-7b011a88a1f0-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:35.159786 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:35.159755    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:35.227857 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:35.223856    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" event={"ID":"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0","Type":"ContainerStarted","Data":"7457678c38a960c83efb6b1f66e025a309f38fb23c1c92b01161890433b0995f"}
Oct 30 19:04:35.227857 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:35.223934    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" event={"ID":"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0","Type":"ContainerStarted","Data":"a91d1970798722c0ee859ea2eb083962a0492a52d71b192a811ecef2432bed84"}
Oct 30 19:04:35.530440 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:35.530345    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"]
Oct 30 19:04:35.537003 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:04:35.536969    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb292c2c3_c427_4e89_8936_7b011a88a1f0.slice/crio-d7947343070266476ae4b2484759c9e43b883ddc90ea7b9e7accbb32222e78ac WatchSource:0}: Error finding container d7947343070266476ae4b2484759c9e43b883ddc90ea7b9e7accbb32222e78ac: Status 404 returned error can't find the container with id d7947343070266476ae4b2484759c9e43b883ddc90ea7b9e7accbb32222e78ac
Oct 30 19:04:36.229531 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:36.229494    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" event={"ID":"b292c2c3-c427-4e89-8936-7b011a88a1f0","Type":"ContainerStarted","Data":"edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec"}
Oct 30 19:04:36.229937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:36.229556    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" event={"ID":"b292c2c3-c427-4e89-8936-7b011a88a1f0","Type":"ContainerStarted","Data":"d7947343070266476ae4b2484759c9e43b883ddc90ea7b9e7accbb32222e78ac"}
Oct 30 19:04:38.240597 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:38.240563    5025 generic.go:356] "Generic (PLEG): container finished" podID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerID="edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec" exitCode=0
Oct 30 19:04:38.241017 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:38.240636    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" event={"ID":"b292c2c3-c427-4e89-8936-7b011a88a1f0","Type":"ContainerDied","Data":"edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec"}
Oct 30 19:04:38.242926 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:38.242901    5025 generic.go:356] "Generic (PLEG): container finished" podID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerID="7457678c38a960c83efb6b1f66e025a309f38fb23c1c92b01161890433b0995f" exitCode=0
Oct 30 19:04:38.243021 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:38.242980    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" event={"ID":"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0","Type":"ContainerDied","Data":"7457678c38a960c83efb6b1f66e025a309f38fb23c1c92b01161890433b0995f"}
Oct 30 19:04:39.249597 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:39.249557    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" event={"ID":"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0","Type":"ContainerStarted","Data":"966c8c1145c2b8ebc6a63d48b0c9a4a56e15e2fc19b4a70cd615c13b80de6ff4"}
Oct 30 19:04:39.251512 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:39.251488    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" event={"ID":"b292c2c3-c427-4e89-8936-7b011a88a1f0","Type":"ContainerStarted","Data":"4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0"}
Oct 30 19:04:39.251700 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:39.251683    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:04:39.386244 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:39.386189    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" podStartSLOduration=2.946210469 podStartE2EDuration="5.386175603s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.439965134s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:39.384387046 +0000 UTC m=+1661.464973253" watchObservedRunningTime="2026-10-30 19:04:39.386175603 +0000 UTC m=+1661.466761809"
Oct 30 19:04:39.514622 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:39.514522    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" podStartSLOduration=3.074853434 podStartE2EDuration="5.51450772s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.439654286s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:04:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:04:39.513690285 +0000 UTC m=+1661.594276511" watchObservedRunningTime="2026-10-30 19:04:39.51450772 +0000 UTC m=+1661.595093926"
Oct 30 19:04:39.528888 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:39.528865    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:44.529006 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:44.528964    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:44.542023 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:44.541994    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:45.301074 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:04:45.301044    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:04:58.084274 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:04:58.084235    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:05:10.268337 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:10.268295    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:05:14.570020 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:14.569983    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq_b292c2c3-c427-4e89-8936-7b011a88a1f0/main/0.log"
Oct 30 19:05:18.145490 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.145455    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"]
Oct 30 19:05:18.145873 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.145655    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" podUID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerName="main" containerID="cri-o://4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0" gracePeriod=30
Oct 30 19:05:18.147199 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.147158    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"]
Oct 30 19:05:18.156668 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:05:18.156629    5025 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="966c8c1145c2b8ebc6a63d48b0c9a4a56e15e2fc19b4a70cd615c13b80de6ff4" cmd=["/bin/sleep","15"]
Oct 30 19:05:18.156773 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.156684    5025 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:05:18.156773 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:05:18.156697    5025 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="main" containerID="cri-o://966c8c1145c2b8ebc6a63d48b0c9a4a56e15e2fc19b4a70cd615c13b80de6ff4"
Oct 30 19:05:18.156773 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.156733    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="main" containerID="cri-o://966c8c1145c2b8ebc6a63d48b0c9a4a56e15e2fc19b4a70cd615c13b80de6ff4" gracePeriod=60
Oct 30 19:05:18.290374 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.290312    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Oct 30 19:05:18.458726 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.458690    5025 generic.go:356] "Generic (PLEG): container finished" podID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerID="966c8c1145c2b8ebc6a63d48b0c9a4a56e15e2fc19b4a70cd615c13b80de6ff4" exitCode=0
Oct 30 19:05:18.458873 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.458780    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" event={"ID":"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0","Type":"ContainerDied","Data":"966c8c1145c2b8ebc6a63d48b0c9a4a56e15e2fc19b4a70cd615c13b80de6ff4"}
Oct 30 19:05:18.517632 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.517604    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:05:18.623252 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623214    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tls-certs\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623485 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623268    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-model-cache\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623485 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623307    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-dshm\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623485 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623347    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tmp-dir\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623695 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623506    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kube-api-access-p6bgm\" (UniqueName: \"kubernetes.io/projected/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kube-api-access-p6bgm\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623695 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623560    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-cabundle-cert\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623695 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623623    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-home\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-home\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623855 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623712    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kserve-provision-location\") pod \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\" (UID: \"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0\") "
Oct 30 19:05:18.623855 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623736    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tmp-dir" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.623960 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.623886    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-model-cache" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.624065 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.624047    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.624126 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.624070    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.624448 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.624266    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-cabundle-cert" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:18.625428 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.625400    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-home" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.625941 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.625910    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tls-certs" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:18.626179 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.626128    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-dshm" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.635997 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.635970    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kube-api-access-p6bgm" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "kube-api-access-p6bgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:05:18.652660 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.652626    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kserve-provision-location" pod "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" (UID: "ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.724632 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.724610    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.724632 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.724634    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6bgm\" (UniqueName: \"kubernetes.io/projected/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kube-api-access-p6bgm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.724771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.724645    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.724771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.724654    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.724771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.724664    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.724771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.724674    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.731287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.731263    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:05:18.825562 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.825535    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-tmp-dir\") pod \"b292c2c3-c427-4e89-8936-7b011a88a1f0\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") "
Oct 30 19:05:18.825737 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.825626    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b292c2c3-c427-4e89-8936-7b011a88a1f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/b292c2c3-c427-4e89-8936-7b011a88a1f0-tls-certs\") pod \"b292c2c3-c427-4e89-8936-7b011a88a1f0\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") "
Oct 30 19:05:18.825737 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.825694    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-kserve-provision-location\") pod \"b292c2c3-c427-4e89-8936-7b011a88a1f0\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") "
Oct 30 19:05:18.825737 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.825731    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b292c2c3-c427-4e89-8936-7b011a88a1f0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b292c2c3-c427-4e89-8936-7b011a88a1f0-cabundle-cert\") pod \"b292c2c3-c427-4e89-8936-7b011a88a1f0\" (UID: \"b292c2c3-c427-4e89-8936-7b011a88a1f0\") "
Oct 30 19:05:18.826089 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.825973    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-tmp-dir" pod "b292c2c3-c427-4e89-8936-7b011a88a1f0" (UID: "b292c2c3-c427-4e89-8936-7b011a88a1f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.826185 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.826123    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b292c2c3-c427-4e89-8936-7b011a88a1f0-cabundle-cert" pod "b292c2c3-c427-4e89-8936-7b011a88a1f0" (UID: "b292c2c3-c427-4e89-8936-7b011a88a1f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:05:18.827753 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.827732    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b292c2c3-c427-4e89-8936-7b011a88a1f0-tls-certs" pod "b292c2c3-c427-4e89-8936-7b011a88a1f0" (UID: "b292c2c3-c427-4e89-8936-7b011a88a1f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:05:18.848346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.848307    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-kserve-provision-location" pod "b292c2c3-c427-4e89-8936-7b011a88a1f0" (UID: "b292c2c3-c427-4e89-8936-7b011a88a1f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:05:18.926468 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.926437    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.926468 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.926464    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b292c2c3-c427-4e89-8936-7b011a88a1f0-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.926468 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.926474    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b292c2c3-c427-4e89-8936-7b011a88a1f0-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:18.926765 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:18.926483    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b292c2c3-c427-4e89-8936-7b011a88a1f0-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:05:19.465612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.465581    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"
Oct 30 19:05:19.465612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.465603    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk" event={"ID":"ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0","Type":"ContainerDied","Data":"a91d1970798722c0ee859ea2eb083962a0492a52d71b192a811ecef2432bed84"}
Oct 30 19:05:19.466148 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.465666    5025 scope.go:122] "RemoveContainer" containerID="966c8c1145c2b8ebc6a63d48b0c9a4a56e15e2fc19b4a70cd615c13b80de6ff4"
Oct 30 19:05:19.467514 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.467487    5025 generic.go:356] "Generic (PLEG): container finished" podID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerID="4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0" exitCode=0
Oct 30 19:05:19.467609 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.467553    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"
Oct 30 19:05:19.467609 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.467572    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" event={"ID":"b292c2c3-c427-4e89-8936-7b011a88a1f0","Type":"ContainerDied","Data":"4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0"}
Oct 30 19:05:19.467679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.467614    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq" event={"ID":"b292c2c3-c427-4e89-8936-7b011a88a1f0","Type":"ContainerDied","Data":"d7947343070266476ae4b2484759c9e43b883ddc90ea7b9e7accbb32222e78ac"}
Oct 30 19:05:19.475018 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.475002    5025 scope.go:122] "RemoveContainer" containerID="7457678c38a960c83efb6b1f66e025a309f38fb23c1c92b01161890433b0995f"
Oct 30 19:05:19.483390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.483373    5025 scope.go:122] "RemoveContainer" containerID="4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0"
Oct 30 19:05:19.491418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.491395    5025 scope.go:122] "RemoveContainer" containerID="edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec"
Oct 30 19:05:19.498767 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.498743    5025 scope.go:122] "RemoveContainer" containerID="4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0"
Oct 30 19:05:19.499037 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:05:19.499020    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0\": container with ID starting with 4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0 not found: ID does not exist" containerID="4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0"
Oct 30 19:05:19.499084 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.499047    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0"} err="failed to get container status \"4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0\": rpc error: code = NotFound desc = could not find container \"4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0\": container with ID starting with 4692d26c61419536b12156d83208df7c53d48dcb0ad680f2b57f0f79a40592f0 not found: ID does not exist"
Oct 30 19:05:19.499084 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.499063    5025 scope.go:122] "RemoveContainer" containerID="edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec"
Oct 30 19:05:19.499281 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:05:19.499267    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec\": container with ID starting with edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec not found: ID does not exist" containerID="edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec"
Oct 30 19:05:19.499337 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.499286    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec"} err="failed to get container status \"edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec\": rpc error: code = NotFound desc = could not find container \"edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec\": container with ID starting with edcdceb315b70a06186007caebaa153d3598400e3207d8e7a1d1fe9bcfeb64ec not found: ID does not exist"
Oct 30 19:05:19.625725 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.625696    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"]
Oct 30 19:05:19.673219 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.673189    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-75768b77f-w7bhk"]
Oct 30 19:05:19.780302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.780271    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"]
Oct 30 19:05:19.788058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:19.788031    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-c7f9f6549-k2pcq"]
Oct 30 19:05:20.056428 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:20.056355    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b292c2c3-c427-4e89-8936-7b011a88a1f0" path="/var/lib/kubelet/pods/b292c2c3-c427-4e89-8936-7b011a88a1f0/volumes"
Oct 30 19:05:20.056841 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:05:20.056826    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" path="/var/lib/kubelet/pods/ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0/volumes"
Oct 30 19:05:28.083036 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:05:28.082990    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:05:58.081660 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:05:58.081622    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:06:14.474958 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.474923    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"]
Oct 30 19:06:14.475380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475297    5025 cpu_manager.go:414] "containerMap: removing container" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="main"
Oct 30 19:06:14.475380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475309    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="main"
Oct 30 19:06:14.475380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475372    5025 cpu_manager.go:414] "containerMap: removing container" podUID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerName="storage-initializer"
Oct 30 19:06:14.475380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475380    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerName="storage-initializer"
Oct 30 19:06:14.475526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475397    5025 cpu_manager.go:414] "containerMap: removing container" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="storage-initializer"
Oct 30 19:06:14.475526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475405    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="storage-initializer"
Oct 30 19:06:14.475526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475418    5025 cpu_manager.go:414] "containerMap: removing container" podUID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerName="main"
Oct 30 19:06:14.475526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475425    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerName="main"
Oct 30 19:06:14.475652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475559    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceba4cf6-f0e4-4a74-9a40-f1838bd4c6e0" containerName="main"
Oct 30 19:06:14.475652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.475576    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b292c2c3-c427-4e89-8936-7b011a88a1f0" containerName="main"
Oct 30 19:06:14.478952 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.478933    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526248 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526216    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dab98f-d0d4-4039-bd63-7058011adc8f-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526399 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526255    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526399 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526279    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dab98f-d0d4-4039-bd63-7058011adc8f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526495 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526422    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-home\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526495 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526487    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxv79\" (UniqueName: \"kubernetes.io/projected/31dab98f-d0d4-4039-bd63-7058011adc8f-kube-api-access-nxv79\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526573 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526522    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-dshm\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526573 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526559    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.526654 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.526637    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-model-cache\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.532703 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.532688    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-5mrjf\""
Oct 30 19:06:14.580160 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.580125    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"seaweedfs-s3-creds\""
Oct 30 19:06:14.580160 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.580132    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Oct 30 19:06:14.583253 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.583230    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"]
Oct 30 19:06:14.619561 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.619534    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:06:14.627588 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.627572    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Oct 30 19:06:14.627888 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.627872    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Oct 30 19:06:14.627931 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.627899    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxv79\" (UniqueName: \"kubernetes.io/projected/31dab98f-d0d4-4039-bd63-7058011adc8f-kube-api-access-nxv79\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.627966 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.627939    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-dshm\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628005 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.627980    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628039 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628023    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-model-cache\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628073 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628064    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dab98f-d0d4-4039-bd63-7058011adc8f-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628112 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628096    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628180 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628133    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dab98f-d0d4-4039-bd63-7058011adc8f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628314 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628265    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-home\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628518 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628488    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-kserve-provision-location\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628619 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628516    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-model-cache\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628686 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628625    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-tmp-dir\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628686 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628636    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-home\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.628921 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.628900    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dab98f-d0d4-4039-bd63-7058011adc8f-cabundle-cert\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.630380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.630361    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-dshm\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.630579 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.630561    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dab98f-d0d4-4039-bd63-7058011adc8f-tls-certs\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.686469 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.686436    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxv79\" (UniqueName: \"kubernetes.io/projected/31dab98f-d0d4-4039-bd63-7058011adc8f-kube-api-access-nxv79\") pod \"tokenizer-template-args-test-kserve-6cd8df4579-tgthn\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:14.794608 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:14.794531    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:15.075231 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.075160    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"]
Oct 30 19:06:15.079967 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.079946    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.134064 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.134035    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.134239 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.134104    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.134239 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.134187    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e78a071-5eda-42c9-b212-a120c8d9dcc0-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.134396 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.134247    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.140676 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.140650    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"]
Oct 30 19:06:15.229134 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.229087    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"]
Oct 30 19:06:15.234994 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.234966    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.235115 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.235038    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.235115 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.235062    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e78a071-5eda-42c9-b212-a120c8d9dcc0-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.235115 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.235105    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.235445 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.235419    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tmp-dir\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.235524 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.235492    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-kserve-provision-location\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.235847 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:06:15.235823    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dab98f_d0d4_4039_bd63_7058011adc8f.slice/crio-b2433b4ced80ada6bbd4018812c855bf5338cf227f81901eff4ec3a7e839299e WatchSource:0}: Error finding container b2433b4ced80ada6bbd4018812c855bf5338cf227f81901eff4ec3a7e839299e: Status 404 returned error can't find the container with id b2433b4ced80ada6bbd4018812c855bf5338cf227f81901eff4ec3a7e839299e
Oct 30 19:06:15.235945 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.235846    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e78a071-5eda-42c9-b212-a120c8d9dcc0-cabundle-cert\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.237823 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.237806    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tls-certs\") pod \"tokenizer-template-args-test-tokenizer-567c9c495b-c74fq\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.395094 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.395057    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:15.726033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.725941    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" event={"ID":"31dab98f-d0d4-4039-bd63-7058011adc8f","Type":"ContainerStarted","Data":"0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c"}
Oct 30 19:06:15.726033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.726009    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" event={"ID":"31dab98f-d0d4-4039-bd63-7058011adc8f","Type":"ContainerStarted","Data":"b2433b4ced80ada6bbd4018812c855bf5338cf227f81901eff4ec3a7e839299e"}
Oct 30 19:06:15.766305 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:15.766278    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"]
Oct 30 19:06:15.772117 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:06:15.772088    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e78a071_5eda_42c9_b212_a120c8d9dcc0.slice/crio-7934d45c5eea0dca5133e024a01abde4a7390ee462c71405b9eb0bba3874a468 WatchSource:0}: Error finding container 7934d45c5eea0dca5133e024a01abde4a7390ee462c71405b9eb0bba3874a468: Status 404 returned error can't find the container with id 7934d45c5eea0dca5133e024a01abde4a7390ee462c71405b9eb0bba3874a468
Oct 30 19:06:16.731903 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:16.731865    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" event={"ID":"7e78a071-5eda-42c9-b212-a120c8d9dcc0","Type":"ContainerStarted","Data":"88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1"}
Oct 30 19:06:16.732296 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:16.731924    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" event={"ID":"7e78a071-5eda-42c9-b212-a120c8d9dcc0","Type":"ContainerStarted","Data":"7934d45c5eea0dca5133e024a01abde4a7390ee462c71405b9eb0bba3874a468"}
Oct 30 19:06:18.744944 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:18.744905    5025 generic.go:356] "Generic (PLEG): container finished" podID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerID="0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c" exitCode=0
Oct 30 19:06:18.745492 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:18.744980    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" event={"ID":"31dab98f-d0d4-4039-bd63-7058011adc8f","Type":"ContainerDied","Data":"0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c"}
Oct 30 19:06:18.747068 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:18.747044    5025 generic.go:356] "Generic (PLEG): container finished" podID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerID="88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1" exitCode=0
Oct 30 19:06:18.747170 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:18.747098    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" event={"ID":"7e78a071-5eda-42c9-b212-a120c8d9dcc0","Type":"ContainerDied","Data":"88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1"}
Oct 30 19:06:19.754061 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:19.754024    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" event={"ID":"7e78a071-5eda-42c9-b212-a120c8d9dcc0","Type":"ContainerStarted","Data":"6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb"}
Oct 30 19:06:19.754542 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:19.754238    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:19.756062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:19.756027    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" event={"ID":"31dab98f-d0d4-4039-bd63-7058011adc8f","Type":"ContainerStarted","Data":"877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a"}
Oct 30 19:06:19.795548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:19.795510    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:19.869711 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:19.869652    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" podStartSLOduration=2.439887039 podStartE2EDuration="4.869637823s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.429750784s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:19.869243054 +0000 UTC m=+1761.949829261" watchObservedRunningTime="2026-10-30 19:06:19.869637823 +0000 UTC m=+1761.950224028"
Oct 30 19:06:19.991131 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:19.991072    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" podStartSLOduration=3.528584991 podStartE2EDuration="5.991059346s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.462474355s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:06:19.989983721 +0000 UTC m=+1762.070569940" watchObservedRunningTime="2026-10-30 19:06:19.991059346 +0000 UTC m=+1762.071645553"
Oct 30 19:06:24.795649 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:24.795615    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:24.808543 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:24.808522    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:25.797722 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:25.797696    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:28.083049 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:28.083010    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:06:50.772642 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:50.772603    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:54.826182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:54.826154    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-0eb24714_tokenizer-template-args-test-tokenizer-567c9c495b-c74fq_7e78a071-5eda-42c9-b212-a120c8d9dcc0/main/0.log"
Oct 30 19:06:58.086969 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:58.086916    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:06:58.153457 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.153426    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"]
Oct 30 19:06:58.155463 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.155385    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"]
Oct 30 19:06:58.155614 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.155561    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" podUID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerName="main" containerID="cri-o://6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb" gracePeriod=30
Oct 30 19:06:58.166208 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:58.166171    5025 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a" cmd=["/bin/sleep","15"]
Oct 30 19:06:58.166356 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.166235    5025 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:06:58.166356 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:58.166253    5025 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="main" containerID="cri-o://877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a"
Oct 30 19:06:58.166356 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.166311    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="main" containerID="cri-o://877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a" gracePeriod=60
Oct 30 19:06:58.319756 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.319727    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:06:58.321231 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.321211    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:06:58.787802 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.787765    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": dial tcp 10.132.0.66:8000: connect: connection refused"
Oct 30 19:06:58.818476 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.818450    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:58.895315 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.895284    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tmp-dir\") pod \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") "
Oct 30 19:06:58.895487 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.895407    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e78a071-5eda-42c9-b212-a120c8d9dcc0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e78a071-5eda-42c9-b212-a120c8d9dcc0-cabundle-cert\") pod \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") "
Oct 30 19:06:58.895487 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.895445    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tls-certs\") pod \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") "
Oct 30 19:06:58.895878 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.895484    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-kserve-provision-location\") pod \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\" (UID: \"7e78a071-5eda-42c9-b212-a120c8d9dcc0\") "
Oct 30 19:06:58.895878 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.895806    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e78a071-5eda-42c9-b212-a120c8d9dcc0-cabundle-cert" pod "7e78a071-5eda-42c9-b212-a120c8d9dcc0" (UID: "7e78a071-5eda-42c9-b212-a120c8d9dcc0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:58.895965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.895905    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tmp-dir" pod "7e78a071-5eda-42c9-b212-a120c8d9dcc0" (UID: "7e78a071-5eda-42c9-b212-a120c8d9dcc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.897813 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.897783    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tls-certs" pod "7e78a071-5eda-42c9-b212-a120c8d9dcc0" (UID: "7e78a071-5eda-42c9-b212-a120c8d9dcc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:58.916824 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.916805    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:58.919941 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.919922    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-kserve-provision-location" pod "7e78a071-5eda-42c9-b212-a120c8d9dcc0" (UID: "7e78a071-5eda-42c9-b212-a120c8d9dcc0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.954540 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.954508    5025 generic.go:356] "Generic (PLEG): container finished" podID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerID="6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb" exitCode=0
Oct 30 19:06:58.954690 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.954567    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"
Oct 30 19:06:58.954690 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.954591    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" event={"ID":"7e78a071-5eda-42c9-b212-a120c8d9dcc0","Type":"ContainerDied","Data":"6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb"}
Oct 30 19:06:58.954690 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.954654    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq" event={"ID":"7e78a071-5eda-42c9-b212-a120c8d9dcc0","Type":"ContainerDied","Data":"7934d45c5eea0dca5133e024a01abde4a7390ee462c71405b9eb0bba3874a468"}
Oct 30 19:06:58.954690 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.954686    5025 scope.go:122] "RemoveContainer" containerID="6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb"
Oct 30 19:06:58.956432 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.956397    5025 generic.go:356] "Generic (PLEG): container finished" podID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerID="877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a" exitCode=0
Oct 30 19:06:58.956530 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.956460    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"
Oct 30 19:06:58.956591 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.956457    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" event={"ID":"31dab98f-d0d4-4039-bd63-7058011adc8f","Type":"ContainerDied","Data":"877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a"}
Oct 30 19:06:58.956640 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.956586    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn" event={"ID":"31dab98f-d0d4-4039-bd63-7058011adc8f","Type":"ContainerDied","Data":"b2433b4ced80ada6bbd4018812c855bf5338cf227f81901eff4ec3a7e839299e"}
Oct 30 19:06:58.965357 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.965339    5025 scope.go:122] "RemoveContainer" containerID="88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1"
Oct 30 19:06:58.973635 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.973617    5025 scope.go:122] "RemoveContainer" containerID="6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb"
Oct 30 19:06:58.973933 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:58.973901    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb\": container with ID starting with 6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb not found: ID does not exist" containerID="6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb"
Oct 30 19:06:58.973987 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.973939    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb"} err="failed to get container status \"6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb\": rpc error: code = NotFound desc = could not find container \"6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb\": container with ID starting with 6f73442633600fe5c14590fc1dfe1cd568778dea5069c5dcfe0d9a3e9eefcceb not found: ID does not exist"
Oct 30 19:06:58.973987 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.973962    5025 scope.go:122] "RemoveContainer" containerID="88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1"
Oct 30 19:06:58.974208 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:58.974186    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1\": container with ID starting with 88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1 not found: ID does not exist" containerID="88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1"
Oct 30 19:06:58.974258 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.974218    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1"} err="failed to get container status \"88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1\": rpc error: code = NotFound desc = could not find container \"88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1\": container with ID starting with 88f85adcfde26f8b9a521aa184f7cc095554443e350a7ba31797ce36849713b1 not found: ID does not exist"
Oct 30 19:06:58.974258 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.974237    5025 scope.go:122] "RemoveContainer" containerID="877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a"
Oct 30 19:06:58.983817 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.983799    5025 scope.go:122] "RemoveContainer" containerID="0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c"
Oct 30 19:06:58.991480 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.991466    5025 scope.go:122] "RemoveContainer" containerID="877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a"
Oct 30 19:06:58.991740 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:58.991722    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a\": container with ID starting with 877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a not found: ID does not exist" containerID="877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a"
Oct 30 19:06:58.991787 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.991746    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a"} err="failed to get container status \"877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a\": rpc error: code = NotFound desc = could not find container \"877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a\": container with ID starting with 877d3840d627a5285a983c1f927618420759f79c561e92e465c044903402c67a not found: ID does not exist"
Oct 30 19:06:58.991787 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.991763    5025 scope.go:122] "RemoveContainer" containerID="0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c"
Oct 30 19:06:58.991986 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:06:58.991971    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c\": container with ID starting with 0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c not found: ID does not exist" containerID="0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c"
Oct 30 19:06:58.991986 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.991988    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c"} err="failed to get container status \"0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c\": rpc error: code = NotFound desc = could not find container \"0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c\": container with ID starting with 0052e45fd3e9854c7c52b97d4111c24f53055d539e7476a34ce7416c4608a84c not found: ID does not exist"
Oct 30 19:06:58.996733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996716    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31dab98f-d0d4-4039-bd63-7058011adc8f-tls-certs\" (UniqueName: \"kubernetes.io/secret/31dab98f-d0d4-4039-bd63-7058011adc8f-tls-certs\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.996808 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996745    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-dshm\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.996808 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996778    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-tmp-dir\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.996889 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996835    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dab98f-d0d4-4039-bd63-7058011adc8f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dab98f-d0d4-4039-bd63-7058011adc8f-cabundle-cert\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.996889 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996867    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-model-cache\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.996995 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996907    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-kserve-provision-location\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.996995 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996944    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dab98f-d0d4-4039-bd63-7058011adc8f-kube-api-access-nxv79\" (UniqueName: \"kubernetes.io/projected/31dab98f-d0d4-4039-bd63-7058011adc8f-kube-api-access-nxv79\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.996995 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.996989    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-home\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-home\") pod \"31dab98f-d0d4-4039-bd63-7058011adc8f\" (UID: \"31dab98f-d0d4-4039-bd63-7058011adc8f\") "
Oct 30 19:06:58.997223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997200    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-model-cache" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.997304 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997204    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dab98f-d0d4-4039-bd63-7058011adc8f-cabundle-cert" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:06:58.997471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997415    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.997471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997441    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7e78a071-5eda-42c9-b212-a120c8d9dcc0-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.997471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997457    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e78a071-5eda-42c9-b212-a120c8d9dcc0-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.997624 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997472    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7e78a071-5eda-42c9-b212-a120c8d9dcc0-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.997624 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997487    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31dab98f-d0d4-4039-bd63-7058011adc8f-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.997624 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997503    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:58.997624 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997437    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-tmp-dir" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.997624 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.997444    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-home" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.998761 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.998743    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31dab98f-d0d4-4039-bd63-7058011adc8f-tls-certs" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:06:58.999261 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.999241    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-dshm" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:58.999304 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:58.999278    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dab98f-d0d4-4039-bd63-7058011adc8f-kube-api-access-nxv79" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "kube-api-access-nxv79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:06:59.018663 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.018640    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-kserve-provision-location" pod "31dab98f-d0d4-4039-bd63-7058011adc8f" (UID: "31dab98f-d0d4-4039-bd63-7058011adc8f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:06:59.098309 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.098271    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31dab98f-d0d4-4039-bd63-7058011adc8f-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.098309 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.098304    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.098736 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.098316    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.098736 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.098356    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.098736 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.098369    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxv79\" (UniqueName: \"kubernetes.io/projected/31dab98f-d0d4-4039-bd63-7058011adc8f-kube-api-access-nxv79\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.098736 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.098382    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31dab98f-d0d4-4039-bd63-7058011adc8f-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:06:59.221051 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.220973    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"]
Oct 30 19:06:59.228328 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.228302    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-tokenizer-567c9c495b-c74fq"]
Oct 30 19:06:59.386044 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.386012    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"]
Oct 30 19:06:59.393404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:06:59.393377    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-6cd8df4579-tgthn"]
Oct 30 19:07:00.055773 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:00.055737    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" path="/var/lib/kubelet/pods/31dab98f-d0d4-4039-bd63-7058011adc8f/volumes"
Oct 30 19:07:00.056213 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:00.056197    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" path="/var/lib/kubelet/pods/7e78a071-5eda-42c9-b212-a120c8d9dcc0/volumes"
Oct 30 19:07:28.080891 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:07:28.080854    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:07:52.707653 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.707573    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"]
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.707945    5025 cpu_manager.go:414] "containerMap: removing container" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="storage-initializer"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.707958    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="storage-initializer"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.707980    5025 cpu_manager.go:414] "containerMap: removing container" podUID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerName="storage-initializer"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.707985    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerName="storage-initializer"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.707998    5025 cpu_manager.go:414] "containerMap: removing container" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="main"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.708003    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="main"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.708014    5025 cpu_manager.go:414] "containerMap: removing container" podUID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerName="main"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.708019    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerName="main"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.708085    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dab98f-d0d4-4039-bd63-7058011adc8f" containerName="main"
Oct 30 19:07:52.708137 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.708092    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e78a071-5eda-42c9-b212-a120c8d9dcc0" containerName="main"
Oct 30 19:07:52.711317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.711296    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:52.763226 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.763196    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-4w8ls\""
Oct 30 19:07:52.789262 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.789225    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm57k\" (UniqueName: \"kubernetes.io/projected/3080eb57-0614-4780-b47d-19aad78a8ac1-kube-api-access-qm57k\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:52.789450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.789276    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3080eb57-0614-4780-b47d-19aad78a8ac1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:52.810568 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.810530    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-ftrrp\""
Oct 30 19:07:52.810733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.810574    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Oct 30 19:07:52.810733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.810584    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Oct 30 19:07:52.810733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.810574    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Oct 30 19:07:52.812544 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.812526    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"]
Oct 30 19:07:52.890346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.890289    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm57k\" (UniqueName: \"kubernetes.io/projected/3080eb57-0614-4780-b47d-19aad78a8ac1-kube-api-access-qm57k\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:52.890346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.890355    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3080eb57-0614-4780-b47d-19aad78a8ac1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:52.892929 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.892907    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3080eb57-0614-4780-b47d-19aad78a8ac1-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:52.946223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:52.946193    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm57k\" (UniqueName: \"kubernetes.io/projected/3080eb57-0614-4780-b47d-19aad78a8ac1-kube-api-access-qm57k\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:53.024413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:53.024334    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:53.398246 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:53.398213    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"]
Oct 30 19:07:53.404133 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:07:53.404105    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3080eb57_0614_4780_b47d_19aad78a8ac1.slice/crio-3ecde9b620ab03b3983a60d243a78410d59fb0ae65cb05702e55183879b37b3c WatchSource:0}: Error finding container 3ecde9b620ab03b3983a60d243a78410d59fb0ae65cb05702e55183879b37b3c: Status 404 returned error can't find the container with id 3ecde9b620ab03b3983a60d243a78410d59fb0ae65cb05702e55183879b37b3c
Oct 30 19:07:54.206944 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:54.206910    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" event={"ID":"3080eb57-0614-4780-b47d-19aad78a8ac1","Type":"ContainerStarted","Data":"084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c"}
Oct 30 19:07:54.207317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:54.206960    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" event={"ID":"3080eb57-0614-4780-b47d-19aad78a8ac1","Type":"ContainerStarted","Data":"3ecde9b620ab03b3983a60d243a78410d59fb0ae65cb05702e55183879b37b3c"}
Oct 30 19:07:54.207317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:07:54.207017    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:07:58.081549 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:07:58.081514    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:08:25.214432 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:25.214396    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:08:25.325540 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:25.325492    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" podStartSLOduration=33.32548045 podStartE2EDuration="33.32548045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:07:54.327588477 +0000 UTC m=+1856.408174683" watchObservedRunningTime="2026-10-30 19:08:25.32548045 +0000 UTC m=+1887.406066656"
Oct 30 19:08:28.082460 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:08:28.082419    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:08:31.895514 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:31.895474    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"]
Oct 30 19:08:35.213998 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:35.213963    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:08:45.214022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:45.213978    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:08:46.915944 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:46.915904    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main" containerID="cri-o://084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c" gracePeriod=45
Oct 30 19:08:55.213597 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:55.213555    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:08:55.213978 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:08:55.213666    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:08:58.086248 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:08:58.086208    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:09:05.213849 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:05.213813    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:09:15.214478 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:15.214393    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:09:17.173900 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.173845    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:09:17.179948 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.179929    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3080eb57-0614-4780-b47d-19aad78a8ac1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3080eb57-0614-4780-b47d-19aad78a8ac1-tls-certs\") pod \"3080eb57-0614-4780-b47d-19aad78a8ac1\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") "
Oct 30 19:09:17.180030 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.180006    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3080eb57-0614-4780-b47d-19aad78a8ac1-kube-api-access-qm57k\" (UniqueName: \"kubernetes.io/projected/3080eb57-0614-4780-b47d-19aad78a8ac1-kube-api-access-qm57k\") pod \"3080eb57-0614-4780-b47d-19aad78a8ac1\" (UID: \"3080eb57-0614-4780-b47d-19aad78a8ac1\") "
Oct 30 19:09:17.181992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.181968    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3080eb57-0614-4780-b47d-19aad78a8ac1-tls-certs" pod "3080eb57-0614-4780-b47d-19aad78a8ac1" (UID: "3080eb57-0614-4780-b47d-19aad78a8ac1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:09:17.182076 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.182039    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3080eb57-0614-4780-b47d-19aad78a8ac1-kube-api-access-qm57k" pod "3080eb57-0614-4780-b47d-19aad78a8ac1" (UID: "3080eb57-0614-4780-b47d-19aad78a8ac1"). InnerVolumeSpecName "kube-api-access-qm57k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:09:17.281562 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.281534    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3080eb57-0614-4780-b47d-19aad78a8ac1-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:17.281562 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.281560    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm57k\" (UniqueName: \"kubernetes.io/projected/3080eb57-0614-4780-b47d-19aad78a8ac1-kube-api-access-qm57k\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:09:17.606526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.606430    5025 generic.go:356] "Generic (PLEG): container finished" podID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerID="084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c" exitCode=0
Oct 30 19:09:17.606526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.606488    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"
Oct 30 19:09:17.606771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.606517    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" event={"ID":"3080eb57-0614-4780-b47d-19aad78a8ac1","Type":"ContainerDied","Data":"084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c"}
Oct 30 19:09:17.606771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.606575    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp" event={"ID":"3080eb57-0614-4780-b47d-19aad78a8ac1","Type":"ContainerDied","Data":"3ecde9b620ab03b3983a60d243a78410d59fb0ae65cb05702e55183879b37b3c"}
Oct 30 19:09:17.606771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.606598    5025 scope.go:122] "RemoveContainer" containerID="084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c"
Oct 30 19:09:17.623363 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.623312    5025 scope.go:122] "RemoveContainer" containerID="084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c"
Oct 30 19:09:17.623693 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:09:17.623668    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c\": container with ID starting with 084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c not found: ID does not exist" containerID="084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c"
Oct 30 19:09:17.623803 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.623698    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c"} err="failed to get container status \"084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c\": rpc error: code = NotFound desc = could not find container \"084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c\": container with ID starting with 084d51f9cfa4cb7e8693170a7679b664d9fc9f865216b47c249477c671b8a90c not found: ID does not exist"
Oct 30 19:09:17.766720 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.766685    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"]
Oct 30 19:09:17.814019 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:17.813988    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-scheqbnbp"]
Oct 30 19:09:18.055785 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:18.055755    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" path="/var/lib/kubelet/pods/3080eb57-0614-4780-b47d-19aad78a8ac1/volumes"
Oct 30 19:09:27.040001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.039962    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"]
Oct 30 19:09:27.040425 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.040394    5025 cpu_manager.go:414] "containerMap: removing container" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main"
Oct 30 19:09:27.040425 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.040408    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main"
Oct 30 19:09:27.040496 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.040478    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3080eb57-0614-4780-b47d-19aad78a8ac1" containerName="main"
Oct 30 19:09:27.043584 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.043559    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.071988 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.071951    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.072156 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.071996    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.072156 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.072142    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.072281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.072208    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.072281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.072243    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8zm5\" (UniqueName: \"kubernetes.io/projected/3bb52863-52b7-469b-839b-e75c0c5d2e85-kube-api-access-q8zm5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.072417 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.072391    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bb52863-52b7-469b-839b-e75c0c5d2e85-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.100200 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.100175    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Oct 30 19:09:27.147618 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.147589    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Oct 30 19:09:27.147618 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.147589    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-rp9gr\""
Oct 30 19:09:27.147953 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.147936    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Oct 30 19:09:27.149987 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.149967    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"]
Oct 30 19:09:27.173774 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.173747    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8zm5\" (UniqueName: \"kubernetes.io/projected/3bb52863-52b7-469b-839b-e75c0c5d2e85-kube-api-access-q8zm5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.173911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.173794    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bb52863-52b7-469b-839b-e75c0c5d2e85-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.173911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.173815    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.173911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.173848    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.174078 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.174018    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.174130 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.174095    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.174355 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.174312    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.174445 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.174415    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.174526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.174504    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.176147 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.176130    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.176377 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.176361    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bb52863-52b7-469b-839b-e75c0c5d2e85-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.229984 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.229946    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8zm5\" (UniqueName: \"kubernetes.io/projected/3bb52863-52b7-469b-839b-e75c0c5d2e85-kube-api-access-q8zm5\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.358975 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.358884    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:27.480085 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.480053    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"]
Oct 30 19:09:27.484654 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.484633    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:27.536155 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.536119    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-5j89q\""
Oct 30 19:09:27.542680 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.542655    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"]
Oct 30 19:09:27.577304 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.577266    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8kr\" (UniqueName: \"kubernetes.io/projected/289895c2-d4b0-4734-827f-2a19c4ca0f71-kube-api-access-7m8kr\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:27.577474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.577380    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289895c2-d4b0-4734-827f-2a19c4ca0f71-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:27.678530 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.678498    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8kr\" (UniqueName: \"kubernetes.io/projected/289895c2-d4b0-4734-827f-2a19c4ca0f71-kube-api-access-7m8kr\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:27.678755 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.678558    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289895c2-d4b0-4734-827f-2a19c4ca0f71-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:27.681054 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.681023    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289895c2-d4b0-4734-827f-2a19c4ca0f71-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:27.728042 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.728007    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"]
Oct 30 19:09:27.733928 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:09:27.733906    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb52863_52b7_469b_839b_e75c0c5d2e85.slice/crio-902c2defc8d4af78c5db061ca3c8460bad8b5af4c23e69280834e68e7dbef503 WatchSource:0}: Error finding container 902c2defc8d4af78c5db061ca3c8460bad8b5af4c23e69280834e68e7dbef503: Status 404 returned error can't find the container with id 902c2defc8d4af78c5db061ca3c8460bad8b5af4c23e69280834e68e7dbef503
Oct 30 19:09:27.739931 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.739906    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8kr\" (UniqueName: \"kubernetes.io/projected/289895c2-d4b0-4734-827f-2a19c4ca0f71-kube-api-access-7m8kr\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:27.797869 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:27.797843    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:28.082251 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:09:28.082166    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:09:28.177351 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.177299    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"]
Oct 30 19:09:28.182977 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:09:28.182948    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289895c2_d4b0_4734_827f_2a19c4ca0f71.slice/crio-f580074b8b6577068849303273ae21a76f9a1656eb570129f7dc873d010fd9a0 WatchSource:0}: Error finding container f580074b8b6577068849303273ae21a76f9a1656eb570129f7dc873d010fd9a0: Status 404 returned error can't find the container with id f580074b8b6577068849303273ae21a76f9a1656eb570129f7dc873d010fd9a0
Oct 30 19:09:28.658700 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.658647    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" event={"ID":"289895c2-d4b0-4734-827f-2a19c4ca0f71","Type":"ContainerStarted","Data":"bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1"}
Oct 30 19:09:28.658887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.658715    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" event={"ID":"289895c2-d4b0-4734-827f-2a19c4ca0f71","Type":"ContainerStarted","Data":"f580074b8b6577068849303273ae21a76f9a1656eb570129f7dc873d010fd9a0"}
Oct 30 19:09:28.658887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.658842    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:09:28.660181 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.660160    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" event={"ID":"3bb52863-52b7-469b-839b-e75c0c5d2e85","Type":"ContainerStarted","Data":"a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57"}
Oct 30 19:09:28.660246 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.660196    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" event={"ID":"3bb52863-52b7-469b-839b-e75c0c5d2e85","Type":"ContainerStarted","Data":"902c2defc8d4af78c5db061ca3c8460bad8b5af4c23e69280834e68e7dbef503"}
Oct 30 19:09:28.769288 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.769220    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" podStartSLOduration=1.769204145 podStartE2EDuration="1.769204145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:28.769058146 +0000 UTC m=+1950.849644355" watchObservedRunningTime="2026-10-30 19:09:28.769204145 +0000 UTC m=+1950.849790349"
Oct 30 19:09:28.881243 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:28.881189    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" podStartSLOduration=2.881171756 podStartE2EDuration="2.881171756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:09:28.879447709 +0000 UTC m=+1950.960033936" watchObservedRunningTime="2026-10-30 19:09:28.881171756 +0000 UTC m=+1950.961757965"
Oct 30 19:09:29.359754 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:29.359714    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:37.359724 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:37.359679    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:37.372407 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:37.372384    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:37.714480 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:37.714442    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:09:58.081136 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:09:58.081099    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:09:59.667060 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:09:59.667031    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:10:09.827172 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.827138    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"]
Oct 30 19:10:09.832233 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.832210    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:09.868467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.868435    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdd9\" (UniqueName: \"kubernetes.io/projected/589477c6-3dd5-40cd-9e42-2a1f17589991-kube-api-access-bkdd9\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:09.868601 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.868565    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589477c6-3dd5-40cd-9e42-2a1f17589991-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:09.898000 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.897973    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-zzt5w\""
Oct 30 19:10:09.930290 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.930257    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Oct 30 19:10:09.932085 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.932064    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"]
Oct 30 19:10:09.969408 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.969378    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdd9\" (UniqueName: \"kubernetes.io/projected/589477c6-3dd5-40cd-9e42-2a1f17589991-kube-api-access-bkdd9\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:09.969569 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.969446    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589477c6-3dd5-40cd-9e42-2a1f17589991-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:09.971942 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:09.971918    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589477c6-3dd5-40cd-9e42-2a1f17589991-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:10.031740 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:10.031698    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdd9\" (UniqueName: \"kubernetes.io/projected/589477c6-3dd5-40cd-9e42-2a1f17589991-kube-api-access-bkdd9\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:10.145442 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:10.145411    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:10.513824 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:10.513746    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"]
Oct 30 19:10:10.519787 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:10:10.519759    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589477c6_3dd5_40cd_9e42_2a1f17589991.slice/crio-a8a3bfd203e30cab6668e85d35f1bbcd138744f83992d9c36fe99e6eaefd5b68 WatchSource:0}: Error finding container a8a3bfd203e30cab6668e85d35f1bbcd138744f83992d9c36fe99e6eaefd5b68: Status 404 returned error can't find the container with id a8a3bfd203e30cab6668e85d35f1bbcd138744f83992d9c36fe99e6eaefd5b68
Oct 30 19:10:10.871646 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:10.871612    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" event={"ID":"589477c6-3dd5-40cd-9e42-2a1f17589991","Type":"ContainerStarted","Data":"e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc"}
Oct 30 19:10:10.872125 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:10.871664    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" event={"ID":"589477c6-3dd5-40cd-9e42-2a1f17589991","Type":"ContainerStarted","Data":"a8a3bfd203e30cab6668e85d35f1bbcd138744f83992d9c36fe99e6eaefd5b68"}
Oct 30 19:10:10.872125 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:10.871695    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:10.983455 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:10.983410    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" podStartSLOduration=1.9833969219999998 podStartE2EDuration="1.983396922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:10:10.98161142 +0000 UTC m=+1993.062197646" watchObservedRunningTime="2026-10-30 19:10:10.983396922 +0000 UTC m=+1993.063983158"
Oct 30 19:10:28.082021 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:10:28.081988    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:10:41.878255 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:41.878181    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:10:48.464102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:48.464063    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"]
Oct 30 19:10:51.878442 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:10:51.878402    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:10:58.082380 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:10:58.082341    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:11:01.878669 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:01.878628    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:03.484381 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:03.484339    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main" containerID="cri-o://e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc" gracePeriod=45
Oct 30 19:11:11.877786 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:11.877748    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:11.878160 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:11.877857    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:11:21.878299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:21.878263    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:28.082785 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:11:28.082747    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:11:31.878406 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:31.878360    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:33.736701 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:33.736679    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:11:33.761776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:33.761718    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589477c6-3dd5-40cd-9e42-2a1f17589991-kube-api-access-bkdd9\" (UniqueName: \"kubernetes.io/projected/589477c6-3dd5-40cd-9e42-2a1f17589991-kube-api-access-bkdd9\") pod \"589477c6-3dd5-40cd-9e42-2a1f17589991\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") "
Oct 30 19:11:33.761776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:33.761773    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589477c6-3dd5-40cd-9e42-2a1f17589991-tls-certs\" (UniqueName: \"kubernetes.io/secret/589477c6-3dd5-40cd-9e42-2a1f17589991-tls-certs\") pod \"589477c6-3dd5-40cd-9e42-2a1f17589991\" (UID: \"589477c6-3dd5-40cd-9e42-2a1f17589991\") "
Oct 30 19:11:33.763919 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:33.763885    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589477c6-3dd5-40cd-9e42-2a1f17589991-tls-certs" pod "589477c6-3dd5-40cd-9e42-2a1f17589991" (UID: "589477c6-3dd5-40cd-9e42-2a1f17589991"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:33.764023 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:33.763945    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589477c6-3dd5-40cd-9e42-2a1f17589991-kube-api-access-bkdd9" pod "589477c6-3dd5-40cd-9e42-2a1f17589991" (UID: "589477c6-3dd5-40cd-9e42-2a1f17589991"). InnerVolumeSpecName "kube-api-access-bkdd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:33.862766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:33.862728    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdd9\" (UniqueName: \"kubernetes.io/projected/589477c6-3dd5-40cd-9e42-2a1f17589991-kube-api-access-bkdd9\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:33.862766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:33.862767    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/589477c6-3dd5-40cd-9e42-2a1f17589991-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:34.271702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.271669    5025 generic.go:356] "Generic (PLEG): container finished" podID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerID="e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc" exitCode=0
Oct 30 19:11:34.271702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.271705    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" event={"ID":"589477c6-3dd5-40cd-9e42-2a1f17589991","Type":"ContainerDied","Data":"e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc"}
Oct 30 19:11:34.271925 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.271723    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"
Oct 30 19:11:34.271925 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.271757    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw" event={"ID":"589477c6-3dd5-40cd-9e42-2a1f17589991","Type":"ContainerDied","Data":"a8a3bfd203e30cab6668e85d35f1bbcd138744f83992d9c36fe99e6eaefd5b68"}
Oct 30 19:11:34.271925 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.271780    5025 scope.go:122] "RemoveContainer" containerID="e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc"
Oct 30 19:11:34.280481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.280460    5025 scope.go:122] "RemoveContainer" containerID="e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc"
Oct 30 19:11:34.280753 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:11:34.280732    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc\": container with ID starting with e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc not found: ID does not exist" containerID="e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc"
Oct 30 19:11:34.280805 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.280760    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc"} err="failed to get container status \"e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc\": rpc error: code = NotFound desc = could not find container \"e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc\": container with ID starting with e3815373741d4ea3fa04459d23e8e2d0d234c6975c7b221faa5052feb0d73bfc not found: ID does not exist"
Oct 30 19:11:34.432532 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.432499    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"]
Oct 30 19:11:34.480025 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:34.479990    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche847mw"]
Oct 30 19:11:36.057132 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:36.057102    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" path="/var/lib/kubelet/pods/589477c6-3dd5-40cd-9e42-2a1f17589991/volumes"
Oct 30 19:11:37.204553 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.204519    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"]
Oct 30 19:11:37.240450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.240418    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"]
Oct 30 19:11:37.249249 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:11:37.249213    5025 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57" cmd=["/bin/sleep","15"]
Oct 30 19:11:37.249423 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.249268    5025 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:11:37.249423 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:11:37.249282    5025 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" podUID="3bb52863-52b7-469b-839b-e75c0c5d2e85" containerName="main" containerID="cri-o://a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57"
Oct 30 19:11:37.249423 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.249336    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" podUID="3bb52863-52b7-469b-839b-e75c0c5d2e85" containerName="main" containerID="cri-o://a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57" gracePeriod=60
Oct 30 19:11:37.599813 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.599790    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:11:37.692937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.692907    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-dshm\") pod \"3bb52863-52b7-469b-839b-e75c0c5d2e85\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") "
Oct 30 19:11:37.692937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.692943    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb52863-52b7-469b-839b-e75c0c5d2e85-tls-certs\" (UniqueName: \"kubernetes.io/secret/3bb52863-52b7-469b-839b-e75c0c5d2e85-tls-certs\") pod \"3bb52863-52b7-469b-839b-e75c0c5d2e85\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") "
Oct 30 19:11:37.693144 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.693006    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-model-cache\") pod \"3bb52863-52b7-469b-839b-e75c0c5d2e85\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") "
Oct 30 19:11:37.693144 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.693047    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb52863-52b7-469b-839b-e75c0c5d2e85-kube-api-access-q8zm5\" (UniqueName: \"kubernetes.io/projected/3bb52863-52b7-469b-839b-e75c0c5d2e85-kube-api-access-q8zm5\") pod \"3bb52863-52b7-469b-839b-e75c0c5d2e85\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") "
Oct 30 19:11:37.693144 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.693074    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-home\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-home\") pod \"3bb52863-52b7-469b-839b-e75c0c5d2e85\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") "
Oct 30 19:11:37.693144 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.693103    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-tmp-dir\") pod \"3bb52863-52b7-469b-839b-e75c0c5d2e85\" (UID: \"3bb52863-52b7-469b-839b-e75c0c5d2e85\") "
Oct 30 19:11:37.693434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.693393    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-home" pod "3bb52863-52b7-469b-839b-e75c0c5d2e85" (UID: "3bb52863-52b7-469b-839b-e75c0c5d2e85"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:37.693489 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.693385    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-model-cache" pod "3bb52863-52b7-469b-839b-e75c0c5d2e85" (UID: "3bb52863-52b7-469b-839b-e75c0c5d2e85"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:37.693489 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.693454    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-tmp-dir" pod "3bb52863-52b7-469b-839b-e75c0c5d2e85" (UID: "3bb52863-52b7-469b-839b-e75c0c5d2e85"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:37.695028 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.695008    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb52863-52b7-469b-839b-e75c0c5d2e85-tls-certs" pod "3bb52863-52b7-469b-839b-e75c0c5d2e85" (UID: "3bb52863-52b7-469b-839b-e75c0c5d2e85"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:11:37.695589 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.695567    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-dshm" pod "3bb52863-52b7-469b-839b-e75c0c5d2e85" (UID: "3bb52863-52b7-469b-839b-e75c0c5d2e85"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:11:37.695589 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.695579    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb52863-52b7-469b-839b-e75c0c5d2e85-kube-api-access-q8zm5" pod "3bb52863-52b7-469b-839b-e75c0c5d2e85" (UID: "3bb52863-52b7-469b-839b-e75c0c5d2e85"). InnerVolumeSpecName "kube-api-access-q8zm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:11:37.794349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.794247    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:37.794349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.794277    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8zm5\" (UniqueName: \"kubernetes.io/projected/3bb52863-52b7-469b-839b-e75c0c5d2e85-kube-api-access-q8zm5\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:37.794349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.794289    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:37.794349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.794301    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:37.794349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.794314    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3bb52863-52b7-469b-839b-e75c0c5d2e85-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:37.794349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:37.794342    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bb52863-52b7-469b-839b-e75c0c5d2e85-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:11:38.293210 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.293176    5025 generic.go:356] "Generic (PLEG): container finished" podID="3bb52863-52b7-469b-839b-e75c0c5d2e85" containerID="a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57" exitCode=0
Oct 30 19:11:38.293674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.293231    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"
Oct 30 19:11:38.293674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.293262    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" event={"ID":"3bb52863-52b7-469b-839b-e75c0c5d2e85","Type":"ContainerDied","Data":"a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57"}
Oct 30 19:11:38.293674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.293315    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j" event={"ID":"3bb52863-52b7-469b-839b-e75c0c5d2e85","Type":"ContainerDied","Data":"902c2defc8d4af78c5db061ca3c8460bad8b5af4c23e69280834e68e7dbef503"}
Oct 30 19:11:38.293674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.293353    5025 scope.go:122] "RemoveContainer" containerID="a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57"
Oct 30 19:11:38.302103 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.302086    5025 scope.go:122] "RemoveContainer" containerID="a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57"
Oct 30 19:11:38.302374 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:11:38.302355    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57\": container with ID starting with a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57 not found: ID does not exist" containerID="a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57"
Oct 30 19:11:38.302438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.302382    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57"} err="failed to get container status \"a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57\": rpc error: code = NotFound desc = could not find container \"a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57\": container with ID starting with a747fca81189593ede76cf188db0fd1e203ba40c3e46f2eeaad89ae12d0d1c57 not found: ID does not exist"
Oct 30 19:11:38.477801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.477773    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"]
Oct 30 19:11:38.525210 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:38.525182    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-5957d558486hw4j"]
Oct 30 19:11:39.667504 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:39.667465    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:40.056693 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:40.056614    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb52863-52b7-469b-839b-e75c0c5d2e85" path="/var/lib/kubelet/pods/3bb52863-52b7-469b-839b-e75c0c5d2e85/volumes"
Oct 30 19:11:49.667234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:49.667194    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:52.224149 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:52.224103    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main" containerID="cri-o://bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1" gracePeriod=45
Oct 30 19:11:58.081669 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:11:58.081634    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:11:58.350369 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:58.350260    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:11:58.352272 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:58.352252    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:11:59.666913 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:59.666873    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:11:59.667279 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:11:59.666980    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:12:09.667699 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:09.667606    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:12:19.667881 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:19.667833    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:12:22.473231 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.473202    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:12:22.499097 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.499017    5025 generic.go:356] "Generic (PLEG): container finished" podID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerID="bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1" exitCode=0
Oct 30 19:12:22.499097 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.499060    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"
Oct 30 19:12:22.499261 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.499093    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" event={"ID":"289895c2-d4b0-4734-827f-2a19c4ca0f71","Type":"ContainerDied","Data":"bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1"}
Oct 30 19:12:22.499261 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.499149    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr" event={"ID":"289895c2-d4b0-4734-827f-2a19c4ca0f71","Type":"ContainerDied","Data":"f580074b8b6577068849303273ae21a76f9a1656eb570129f7dc873d010fd9a0"}
Oct 30 19:12:22.499261 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.499172    5025 scope.go:122] "RemoveContainer" containerID="bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1"
Oct 30 19:12:22.507256 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.507239    5025 scope.go:122] "RemoveContainer" containerID="bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1"
Oct 30 19:12:22.507530 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:12:22.507511    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1\": container with ID starting with bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1 not found: ID does not exist" containerID="bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1"
Oct 30 19:12:22.507603 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.507538    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1"} err="failed to get container status \"bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1\": rpc error: code = NotFound desc = could not find container \"bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1\": container with ID starting with bf6c50268e6bbe6a919186b8831882d7a8511e6a4ed1c6514660bb068fe573c1 not found: ID does not exist"
Oct 30 19:12:22.598175 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.598148    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/289895c2-d4b0-4734-827f-2a19c4ca0f71-kube-api-access-7m8kr\" (UniqueName: \"kubernetes.io/projected/289895c2-d4b0-4734-827f-2a19c4ca0f71-kube-api-access-7m8kr\") pod \"289895c2-d4b0-4734-827f-2a19c4ca0f71\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") "
Oct 30 19:12:22.598296 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.598243    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/289895c2-d4b0-4734-827f-2a19c4ca0f71-tls-certs\" (UniqueName: \"kubernetes.io/secret/289895c2-d4b0-4734-827f-2a19c4ca0f71-tls-certs\") pod \"289895c2-d4b0-4734-827f-2a19c4ca0f71\" (UID: \"289895c2-d4b0-4734-827f-2a19c4ca0f71\") "
Oct 30 19:12:22.600263 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.600237    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/289895c2-d4b0-4734-827f-2a19c4ca0f71-tls-certs" pod "289895c2-d4b0-4734-827f-2a19c4ca0f71" (UID: "289895c2-d4b0-4734-827f-2a19c4ca0f71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:12:22.600374 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.600313    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289895c2-d4b0-4734-827f-2a19c4ca0f71-kube-api-access-7m8kr" pod "289895c2-d4b0-4734-827f-2a19c4ca0f71" (UID: "289895c2-d4b0-4734-827f-2a19c4ca0f71"). InnerVolumeSpecName "kube-api-access-7m8kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:12:22.699133 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.699093    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289895c2-d4b0-4734-827f-2a19c4ca0f71-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.699133 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.699130    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m8kr\" (UniqueName: \"kubernetes.io/projected/289895c2-d4b0-4734-827f-2a19c4ca0f71-kube-api-access-7m8kr\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:12:22.915790 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.915754    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"]
Oct 30 19:12:22.922689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:22.922661    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schew7nzr"]
Oct 30 19:12:24.056126 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:24.056094    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" path="/var/lib/kubelet/pods/289895c2-d4b0-4734-827f-2a19c4ca0f71/volumes"
Oct 30 19:12:28.083871 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:12:28.083828    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:12:35.168440 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.168406    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"]
Oct 30 19:12:35.168971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.168955    5025 cpu_manager.go:414] "containerMap: removing container" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main"
Oct 30 19:12:35.169034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.168974    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main"
Oct 30 19:12:35.169034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.168997    5025 cpu_manager.go:414] "containerMap: removing container" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main"
Oct 30 19:12:35.169034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.169005    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main"
Oct 30 19:12:35.169128 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.169050    5025 cpu_manager.go:414] "containerMap: removing container" podUID="3bb52863-52b7-469b-839b-e75c0c5d2e85" containerName="main"
Oct 30 19:12:35.169128 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.169058    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb52863-52b7-469b-839b-e75c0c5d2e85" containerName="main"
Oct 30 19:12:35.169191 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.169154    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb52863-52b7-469b-839b-e75c0c5d2e85" containerName="main"
Oct 30 19:12:35.169191 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.169168    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="589477c6-3dd5-40cd-9e42-2a1f17589991" containerName="main"
Oct 30 19:12:35.169191 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.169179    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="289895c2-d4b0-4734-827f-2a19c4ca0f71" containerName="main"
Oct 30 19:12:35.173973 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.173952    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.207713 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.207626    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.207842 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.207759    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.207842 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.207805    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll27x\" (UniqueName: \"kubernetes.io/projected/da37d6d1-b127-4591-97df-cc4660dbf971-kube-api-access-ll27x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.207842 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.207836    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da37d6d1-b127-4591-97df-cc4660dbf971-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.207957 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.207941    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.207995 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.207975    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.273060 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.273031    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-cx4x8\""
Oct 30 19:12:35.273214 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.273076    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Oct 30 19:12:35.273214 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.273079    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Oct 30 19:12:35.273214 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.273104    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Oct 30 19:12:35.274940 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.274920    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"]
Oct 30 19:12:35.309208 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309181    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309364 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309218    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309364 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309272    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309364 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309291    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309364 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309333    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll27x\" (UniqueName: \"kubernetes.io/projected/da37d6d1-b127-4591-97df-cc4660dbf971-kube-api-access-ll27x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309571 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309390    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da37d6d1-b127-4591-97df-cc4660dbf971-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309627 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309603    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309701 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309675    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.309804 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.309780    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.311640 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.311619    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.311852 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.311834    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da37d6d1-b127-4591-97df-cc4660dbf971-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.372556 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.372524    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll27x\" (UniqueName: \"kubernetes.io/projected/da37d6d1-b127-4591-97df-cc4660dbf971-kube-api-access-ll27x\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.487887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.487798    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:35.594153 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.594115    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"]
Oct 30 19:12:35.597985 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.597961    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:35.649182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.649157    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-mcksv\""
Oct 30 19:12:35.660745 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.660719    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"]
Oct 30 19:12:35.712994 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.712962    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86724e6-a8bd-4f01-9be6-3434c3eae728-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:35.713158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.713136    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clb9g\" (UniqueName: \"kubernetes.io/projected/a86724e6-a8bd-4f01-9be6-3434c3eae728-kube-api-access-clb9g\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:35.813733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.813641    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clb9g\" (UniqueName: \"kubernetes.io/projected/a86724e6-a8bd-4f01-9be6-3434c3eae728-kube-api-access-clb9g\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:35.813733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.813707    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86724e6-a8bd-4f01-9be6-3434c3eae728-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:35.816271 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.816251    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86724e6-a8bd-4f01-9be6-3434c3eae728-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:35.854949 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.854921    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"]
Oct 30 19:12:35.860964 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:12:35.860940    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda37d6d1_b127_4591_97df_cc4660dbf971.slice/crio-ce34d55af74a0a62975f1d9d4f92c73a67f8a0eae366bcbeacdf2d1743b267cb WatchSource:0}: Error finding container ce34d55af74a0a62975f1d9d4f92c73a67f8a0eae366bcbeacdf2d1743b267cb: Status 404 returned error can't find the container with id ce34d55af74a0a62975f1d9d4f92c73a67f8a0eae366bcbeacdf2d1743b267cb
Oct 30 19:12:35.873502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.873477    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clb9g\" (UniqueName: \"kubernetes.io/projected/a86724e6-a8bd-4f01-9be6-3434c3eae728-kube-api-access-clb9g\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:35.910463 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:35.910441    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:36.283234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.283203    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"]
Oct 30 19:12:36.289805 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:12:36.289772    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86724e6_a8bd_4f01_9be6_3434c3eae728.slice/crio-473a454e6e443b5919ac1e1a2ece57a3f909711e47b824a8d5456b550057f906 WatchSource:0}: Error finding container 473a454e6e443b5919ac1e1a2ece57a3f909711e47b824a8d5456b550057f906: Status 404 returned error can't find the container with id 473a454e6e443b5919ac1e1a2ece57a3f909711e47b824a8d5456b550057f906
Oct 30 19:12:36.564510 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.564397    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" event={"ID":"a86724e6-a8bd-4f01-9be6-3434c3eae728","Type":"ContainerStarted","Data":"fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d"}
Oct 30 19:12:36.564510 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.564506    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:12:36.564778 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.564535    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" event={"ID":"a86724e6-a8bd-4f01-9be6-3434c3eae728","Type":"ContainerStarted","Data":"473a454e6e443b5919ac1e1a2ece57a3f909711e47b824a8d5456b550057f906"}
Oct 30 19:12:36.566033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.566003    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" event={"ID":"da37d6d1-b127-4591-97df-cc4660dbf971","Type":"ContainerStarted","Data":"c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc"}
Oct 30 19:12:36.566135 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.566049    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" event={"ID":"da37d6d1-b127-4591-97df-cc4660dbf971","Type":"ContainerStarted","Data":"ce34d55af74a0a62975f1d9d4f92c73a67f8a0eae366bcbeacdf2d1743b267cb"}
Oct 30 19:12:36.675109 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.675064    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" podStartSLOduration=1.675051005 podStartE2EDuration="1.675051005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:36.674360635 +0000 UTC m=+2138.754946843" watchObservedRunningTime="2026-10-30 19:12:36.675051005 +0000 UTC m=+2138.755637205"
Oct 30 19:12:36.785080 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:36.785031    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" podStartSLOduration=1.785018421 podStartE2EDuration="1.785018421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:12:36.784505868 +0000 UTC m=+2138.865092075" watchObservedRunningTime="2026-10-30 19:12:36.785018421 +0000 UTC m=+2138.865604627"
Oct 30 19:12:37.488028 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:37.487998    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:45.488122 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:45.488083    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:45.500924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:45.500900    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:45.661063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:12:45.661035    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:12:58.086023 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:12:58.085986    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:13:07.572965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:07.572931    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:13:18.174344 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.174288    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"]
Oct 30 19:13:18.179415 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.179394    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.232594 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.232561    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"]
Oct 30 19:13:18.277966 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.277939    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Oct 30 19:13:18.278132 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.277979    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-rf2nf\""
Oct 30 19:13:18.295759 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.295728    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25295315-4886-4eb0-b5db-5c07e1b6eb29-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.295889 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.295783    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6hk9\" (UniqueName: \"kubernetes.io/projected/25295315-4886-4eb0-b5db-5c07e1b6eb29-kube-api-access-s6hk9\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.396870 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.396838    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25295315-4886-4eb0-b5db-5c07e1b6eb29-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.397030 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.396892    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6hk9\" (UniqueName: \"kubernetes.io/projected/25295315-4886-4eb0-b5db-5c07e1b6eb29-kube-api-access-s6hk9\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.399331 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.399299    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25295315-4886-4eb0-b5db-5c07e1b6eb29-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.453199 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.453133    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6hk9\" (UniqueName: \"kubernetes.io/projected/25295315-4886-4eb0-b5db-5c07e1b6eb29-kube-api-access-s6hk9\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.491546 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.491518    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:18.861953 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:18.861879    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"]
Oct 30 19:13:18.867300 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:13:18.867268    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25295315_4886_4eb0_b5db_5c07e1b6eb29.slice/crio-fc14fa7719df4a810345d1e191d9198d939bd9bde6f1bdf94eca609eb96f1e3b WatchSource:0}: Error finding container fc14fa7719df4a810345d1e191d9198d939bd9bde6f1bdf94eca609eb96f1e3b: Status 404 returned error can't find the container with id fc14fa7719df4a810345d1e191d9198d939bd9bde6f1bdf94eca609eb96f1e3b
Oct 30 19:13:19.814465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:19.814432    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" event={"ID":"25295315-4886-4eb0-b5db-5c07e1b6eb29","Type":"ContainerStarted","Data":"f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996"}
Oct 30 19:13:19.814859 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:19.814484    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" event={"ID":"25295315-4886-4eb0-b5db-5c07e1b6eb29","Type":"ContainerStarted","Data":"fc14fa7719df4a810345d1e191d9198d939bd9bde6f1bdf94eca609eb96f1e3b"}
Oct 30 19:13:19.814859 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:19.814518    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:19.925743 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:19.925689    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" podStartSLOduration=1.9256694429999999 podStartE2EDuration="1.925669443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:13:19.924831209 +0000 UTC m=+2182.005417416" watchObservedRunningTime="2026-10-30 19:13:19.925669443 +0000 UTC m=+2182.006255653"
Oct 30 19:13:28.082717 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:13:28.082617    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:13:50.821381 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:50.821295    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:13:57.323345 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:13:57.287573    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"]
Oct 30 19:13:58.082482 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:13:58.082448    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:14:00.821214 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:00.821176    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:14:10.821258 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:10.821210    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:14:12.375639 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:12.375600    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main" containerID="cri-o://f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996" gracePeriod=45
Oct 30 19:14:20.821249 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:20.821209    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:14:20.821787 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:20.821348    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:14:28.081785 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:14:28.081745    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:14:30.821649 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:30.821613    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:14:40.821357 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:40.821302    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:14:42.623314 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:42.623287    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:14:42.686455 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:42.686384    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25295315-4886-4eb0-b5db-5c07e1b6eb29-tls-certs\" (UniqueName: \"kubernetes.io/secret/25295315-4886-4eb0-b5db-5c07e1b6eb29-tls-certs\") pod \"25295315-4886-4eb0-b5db-5c07e1b6eb29\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") "
Oct 30 19:14:42.686455 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:42.686433    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25295315-4886-4eb0-b5db-5c07e1b6eb29-kube-api-access-s6hk9\" (UniqueName: \"kubernetes.io/projected/25295315-4886-4eb0-b5db-5c07e1b6eb29-kube-api-access-s6hk9\") pod \"25295315-4886-4eb0-b5db-5c07e1b6eb29\" (UID: \"25295315-4886-4eb0-b5db-5c07e1b6eb29\") "
Oct 30 19:14:42.688636 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:42.688608    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25295315-4886-4eb0-b5db-5c07e1b6eb29-kube-api-access-s6hk9" pod "25295315-4886-4eb0-b5db-5c07e1b6eb29" (UID: "25295315-4886-4eb0-b5db-5c07e1b6eb29"). InnerVolumeSpecName "kube-api-access-s6hk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:42.688757 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:42.688633    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25295315-4886-4eb0-b5db-5c07e1b6eb29-tls-certs" pod "25295315-4886-4eb0-b5db-5c07e1b6eb29" (UID: "25295315-4886-4eb0-b5db-5c07e1b6eb29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:42.787971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:42.787940    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25295315-4886-4eb0-b5db-5c07e1b6eb29-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:42.787971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:42.787967    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6hk9\" (UniqueName: \"kubernetes.io/projected/25295315-4886-4eb0-b5db-5c07e1b6eb29-kube-api-access-s6hk9\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:43.223884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.223851    5025 generic.go:356] "Generic (PLEG): container finished" podID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerID="f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996" exitCode=0
Oct 30 19:14:43.224072 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.223898    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"
Oct 30 19:14:43.224072 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.223921    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" event={"ID":"25295315-4886-4eb0-b5db-5c07e1b6eb29","Type":"ContainerDied","Data":"f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996"}
Oct 30 19:14:43.224072 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.223963    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k" event={"ID":"25295315-4886-4eb0-b5db-5c07e1b6eb29","Type":"ContainerDied","Data":"fc14fa7719df4a810345d1e191d9198d939bd9bde6f1bdf94eca609eb96f1e3b"}
Oct 30 19:14:43.224072 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.223987    5025 scope.go:122] "RemoveContainer" containerID="f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996"
Oct 30 19:14:43.233596 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.233577    5025 scope.go:122] "RemoveContainer" containerID="f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996"
Oct 30 19:14:43.233848 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:14:43.233826    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996\": container with ID starting with f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996 not found: ID does not exist" containerID="f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996"
Oct 30 19:14:43.233902 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.233856    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996"} err="failed to get container status \"f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996\": rpc error: code = NotFound desc = could not find container \"f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996\": container with ID starting with f845ad1fd8e25a336c6a288712de99105ba8cae5df524bfb64cebcf29fb10996 not found: ID does not exist"
Oct 30 19:14:43.387035 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.387002    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"]
Oct 30 19:14:43.390053 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:43.390029    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scherfm7k"]
Oct 30 19:14:44.056722 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:44.056681    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" path="/var/lib/kubelet/pods/25295315-4886-4eb0-b5db-5c07e1b6eb29/volumes"
Oct 30 19:14:46.530808 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:46.530773    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"]
Oct 30 19:14:46.540616 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:14:46.540573    5025 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" containerID="c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc" cmd=["/bin/sleep","15"]
Oct 30 19:14:46.540777 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:46.540640    5025 handlers.go:85] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" message="" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1"
Oct 30 19:14:46.540777 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:14:46.540658    5025 kuberuntime_container.go:780] "PreStop hook failed" err="rpc error: code = Unknown desc = command error: , stdout: , stderr: , exit code -1" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" podUID="da37d6d1-b127-4591-97df-cc4660dbf971" containerName="main" containerID="cri-o://c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc"
Oct 30 19:14:46.540777 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:46.540716    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" podUID="da37d6d1-b127-4591-97df-cc4660dbf971" containerName="main" containerID="cri-o://c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc" gracePeriod=60
Oct 30 19:14:46.548916 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:46.548893    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"]
Oct 30 19:14:46.650926 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:46.650881    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" podUID="da37d6d1-b127-4591-97df-cc4660dbf971" containerName="main" probeResult="failure" output="Get \"https://10.132.0.72:8000/health\": dial tcp 10.132.0.72:8000: connect: connection refused"
Oct 30 19:14:46.904431 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:46.904411    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:14:47.026308 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026260    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-tmp-dir\") pod \"da37d6d1-b127-4591-97df-cc4660dbf971\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") "
Oct 30 19:14:47.026308 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026305    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da37d6d1-b127-4591-97df-cc4660dbf971-tls-certs\" (UniqueName: \"kubernetes.io/secret/da37d6d1-b127-4591-97df-cc4660dbf971-tls-certs\") pod \"da37d6d1-b127-4591-97df-cc4660dbf971\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") "
Oct 30 19:14:47.027058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026412    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-home\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-home\") pod \"da37d6d1-b127-4591-97df-cc4660dbf971\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") "
Oct 30 19:14:47.027058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026470    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da37d6d1-b127-4591-97df-cc4660dbf971-kube-api-access-ll27x\" (UniqueName: \"kubernetes.io/projected/da37d6d1-b127-4591-97df-cc4660dbf971-kube-api-access-ll27x\") pod \"da37d6d1-b127-4591-97df-cc4660dbf971\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") "
Oct 30 19:14:47.027058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026499    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-dshm\") pod \"da37d6d1-b127-4591-97df-cc4660dbf971\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") "
Oct 30 19:14:47.027058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026579    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-model-cache\") pod \"da37d6d1-b127-4591-97df-cc4660dbf971\" (UID: \"da37d6d1-b127-4591-97df-cc4660dbf971\") "
Oct 30 19:14:47.027058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026920    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-home" pod "da37d6d1-b127-4591-97df-cc4660dbf971" (UID: "da37d6d1-b127-4591-97df-cc4660dbf971"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:47.027058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.026948    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-tmp-dir" pod "da37d6d1-b127-4591-97df-cc4660dbf971" (UID: "da37d6d1-b127-4591-97df-cc4660dbf971"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:47.027058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.027072    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-model-cache" pod "da37d6d1-b127-4591-97df-cc4660dbf971" (UID: "da37d6d1-b127-4591-97df-cc4660dbf971"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:47.029665 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.029154    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-dshm" pod "da37d6d1-b127-4591-97df-cc4660dbf971" (UID: "da37d6d1-b127-4591-97df-cc4660dbf971"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:14:47.029665 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.029174    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da37d6d1-b127-4591-97df-cc4660dbf971-kube-api-access-ll27x" pod "da37d6d1-b127-4591-97df-cc4660dbf971" (UID: "da37d6d1-b127-4591-97df-cc4660dbf971"). InnerVolumeSpecName "kube-api-access-ll27x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:14:47.029665 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.029219    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da37d6d1-b127-4591-97df-cc4660dbf971-tls-certs" pod "da37d6d1-b127-4591-97df-cc4660dbf971" (UID: "da37d6d1-b127-4591-97df-cc4660dbf971"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:14:47.128229 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.128197    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:47.128229 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.128230    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da37d6d1-b127-4591-97df-cc4660dbf971-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:47.128467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.128245    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:47.128467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.128261    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll27x\" (UniqueName: \"kubernetes.io/projected/da37d6d1-b127-4591-97df-cc4660dbf971-kube-api-access-ll27x\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:47.128467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.128275    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:47.128467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.128289    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da37d6d1-b127-4591-97df-cc4660dbf971-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:14:47.245624 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.245594    5025 generic.go:356] "Generic (PLEG): container finished" podID="da37d6d1-b127-4591-97df-cc4660dbf971" containerID="c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc" exitCode=0
Oct 30 19:14:47.245789 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.245641    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"
Oct 30 19:14:47.245789 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.245687    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" event={"ID":"da37d6d1-b127-4591-97df-cc4660dbf971","Type":"ContainerDied","Data":"c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc"}
Oct 30 19:14:47.245789 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.245741    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl" event={"ID":"da37d6d1-b127-4591-97df-cc4660dbf971","Type":"ContainerDied","Data":"ce34d55af74a0a62975f1d9d4f92c73a67f8a0eae366bcbeacdf2d1743b267cb"}
Oct 30 19:14:47.245789 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.245763    5025 scope.go:122] "RemoveContainer" containerID="c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc"
Oct 30 19:14:47.257466 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.257446    5025 scope.go:122] "RemoveContainer" containerID="c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc"
Oct 30 19:14:47.257721 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:14:47.257703    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc\": container with ID starting with c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc not found: ID does not exist" containerID="c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc"
Oct 30 19:14:47.257791 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.257729    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc"} err="failed to get container status \"c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc\": rpc error: code = NotFound desc = could not find container \"c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc\": container with ID starting with c967078ca7fe7464ac8fdb54c957d9c4db91120cdb8ad61970457c634494e6fc not found: ID does not exist"
Oct 30 19:14:47.405766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.405696    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"]
Oct 30 19:14:47.411069 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.411046    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-f966769b4-5p7wl"]
Oct 30 19:14:47.572936 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:47.572895    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:14:48.056155 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:48.056123    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da37d6d1-b127-4591-97df-cc4660dbf971" path="/var/lib/kubelet/pods/da37d6d1-b127-4591-97df-cc4660dbf971/volumes"
Oct 30 19:14:57.573240 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:14:57.573200    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:14:58.081791 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:14:58.081760    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:15:01.568043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:01.567983    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main" containerID="cri-o://fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d" gracePeriod=45
Oct 30 19:15:07.573190 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:07.573104    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:07.573570 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:07.573208    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:15:17.573222 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:17.573178    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:27.572516 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:27.572466    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:15:28.081555 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:15:28.081519    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:15:31.828204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:31.828146    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:15:31.925302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:31.925266    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a86724e6-a8bd-4f01-9be6-3434c3eae728-tls-certs\" (UniqueName: \"kubernetes.io/secret/a86724e6-a8bd-4f01-9be6-3434c3eae728-tls-certs\") pod \"a86724e6-a8bd-4f01-9be6-3434c3eae728\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") "
Oct 30 19:15:31.925500 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:31.925345    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a86724e6-a8bd-4f01-9be6-3434c3eae728-kube-api-access-clb9g\" (UniqueName: \"kubernetes.io/projected/a86724e6-a8bd-4f01-9be6-3434c3eae728-kube-api-access-clb9g\") pod \"a86724e6-a8bd-4f01-9be6-3434c3eae728\" (UID: \"a86724e6-a8bd-4f01-9be6-3434c3eae728\") "
Oct 30 19:15:31.927637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:31.927609    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86724e6-a8bd-4f01-9be6-3434c3eae728-tls-certs" pod "a86724e6-a8bd-4f01-9be6-3434c3eae728" (UID: "a86724e6-a8bd-4f01-9be6-3434c3eae728"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:15:31.927765 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:31.927610    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86724e6-a8bd-4f01-9be6-3434c3eae728-kube-api-access-clb9g" pod "a86724e6-a8bd-4f01-9be6-3434c3eae728" (UID: "a86724e6-a8bd-4f01-9be6-3434c3eae728"). InnerVolumeSpecName "kube-api-access-clb9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:32.026721 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.026686    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a86724e6-a8bd-4f01-9be6-3434c3eae728-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:32.026721 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.026715    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clb9g\" (UniqueName: \"kubernetes.io/projected/a86724e6-a8bd-4f01-9be6-3434c3eae728-kube-api-access-clb9g\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:32.452437 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.452405    5025 generic.go:356] "Generic (PLEG): container finished" podID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerID="fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d" exitCode=0
Oct 30 19:15:32.452622 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.452453    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"
Oct 30 19:15:32.452622 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.452454    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" event={"ID":"a86724e6-a8bd-4f01-9be6-3434c3eae728","Type":"ContainerDied","Data":"fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d"}
Oct 30 19:15:32.452622 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.452588    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg" event={"ID":"a86724e6-a8bd-4f01-9be6-3434c3eae728","Type":"ContainerDied","Data":"473a454e6e443b5919ac1e1a2ece57a3f909711e47b824a8d5456b550057f906"}
Oct 30 19:15:32.452622 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.452617    5025 scope.go:122] "RemoveContainer" containerID="fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d"
Oct 30 19:15:32.462304 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.462285    5025 scope.go:122] "RemoveContainer" containerID="fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d"
Oct 30 19:15:32.462578 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:15:32.462558    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d\": container with ID starting with fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d not found: ID does not exist" containerID="fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d"
Oct 30 19:15:32.462654 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.462586    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d"} err="failed to get container status \"fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d\": rpc error: code = NotFound desc = could not find container \"fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d\": container with ID starting with fe44423fb63063a0100cb4644e290d9bcd995f335300c65e714d9d9ea66b345d not found: ID does not exist"
Oct 30 19:15:32.614439 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.614400    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"]
Oct 30 19:15:32.659484 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:32.659448    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schernxxg"]
Oct 30 19:15:34.057097 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:34.057065    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" path="/var/lib/kubelet/pods/a86724e6-a8bd-4f01-9be6-3434c3eae728/volumes"
Oct 30 19:15:44.081492 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:44.081461    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"]
Oct 30 19:15:44.081928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:44.081652    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy" containerID="cri-o://22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80" gracePeriod=30
Oct 30 19:15:44.129065 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:44.129034    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"]
Oct 30 19:15:44.129252 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:44.129233    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy" containerID="cri-o://4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7" gracePeriod=30
Oct 30 19:15:45.332765 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:45.332716    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:15:45.333295 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:45.332784    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:15:48.628600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.628567    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"]
Oct 30 19:15:48.628991 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.628974    5025 cpu_manager.go:414] "containerMap: removing container" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main"
Oct 30 19:15:48.628991 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.628987    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main"
Oct 30 19:15:48.629063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.629005    5025 cpu_manager.go:414] "containerMap: removing container" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main"
Oct 30 19:15:48.629063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.629010    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main"
Oct 30 19:15:48.629063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.629028    5025 cpu_manager.go:414] "containerMap: removing container" podUID="da37d6d1-b127-4591-97df-cc4660dbf971" containerName="main"
Oct 30 19:15:48.629063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.629034    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da37d6d1-b127-4591-97df-cc4660dbf971" containerName="main"
Oct 30 19:15:48.629182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.629092    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="da37d6d1-b127-4591-97df-cc4660dbf971" containerName="main"
Oct 30 19:15:48.629182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.629101    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="25295315-4886-4eb0-b5db-5c07e1b6eb29" containerName="main"
Oct 30 19:15:48.629182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.629108    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="a86724e6-a8bd-4f01-9be6-3434c3eae728" containerName="main"
Oct 30 19:15:48.633716 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.633697    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.682538 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.682503    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"]
Oct 30 19:15:48.727246 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.727219    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Oct 30 19:15:48.727451 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.727272    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Oct 30 19:15:48.727451 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.727365    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-zfz2x\""
Oct 30 19:15:48.727451 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.727368    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Oct 30 19:15:48.727451 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.727385    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:15:48.727617 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.727453    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Oct 30 19:15:48.779745 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.779715    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.779878 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.779750    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.779878 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.779780    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.780001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.779910    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.780063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.780014    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxr6x\" (UniqueName: \"kubernetes.io/projected/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kube-api-access-pxr6x\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.780118 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.780072    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.780118 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.780111    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.780213 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.780194    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.880835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880753    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.880835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880788    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.880835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880810    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.880835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880834    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880860    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880900    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxr6x\" (UniqueName: \"kubernetes.io/projected/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kube-api-access-pxr6x\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880932    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.880962    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881407 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.881220    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881407 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.881372    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881521 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.881403    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881521 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.881476    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.881765 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.881747    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.883346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.883295    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.883598 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.883581    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.936907 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.936874    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxr6x\" (UniqueName: \"kubernetes.io/projected/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kube-api-access-pxr6x\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:48.949739 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:48.949717    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:49.072418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.072387    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"]
Oct 30 19:15:49.076116 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.076095    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.129178 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.129119    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-hskl8\""
Oct 30 19:15:49.130668 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.130633    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"]
Oct 30 19:15:49.183631 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.183600    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz5bm\" (UniqueName: \"kubernetes.io/projected/e79e3736-b546-400c-8137-2d07449be330-kube-api-access-tz5bm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.183840 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.183826    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79e3736-b546-400c-8137-2d07449be330-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.285422 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.285376    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz5bm\" (UniqueName: \"kubernetes.io/projected/e79e3736-b546-400c-8137-2d07449be330-kube-api-access-tz5bm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.285600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.285490    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79e3736-b546-400c-8137-2d07449be330-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.288305 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.288251    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79e3736-b546-400c-8137-2d07449be330-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.342627 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.342595    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz5bm\" (UniqueName: \"kubernetes.io/projected/e79e3736-b546-400c-8137-2d07449be330-kube-api-access-tz5bm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.361379 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.361334    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 19:15:49.375084 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.375048    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"]
Oct 30 19:15:49.382221 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:15:49.382195    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e290002_fa1d_43fa_b14a_2ab7c1f36ef0.slice/crio-b0c3402a226e943fa4265c7f9d06d74ef4fb57fa03b36a57fbd60521af5b148e WatchSource:0}: Error finding container b0c3402a226e943fa4265c7f9d06d74ef4fb57fa03b36a57fbd60521af5b148e: Status 404 returned error can't find the container with id b0c3402a226e943fa4265c7f9d06d74ef4fb57fa03b36a57fbd60521af5b148e
Oct 30 19:15:49.390378 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.390354    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:49.398729 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.398709    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 19:15:49.487513 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487483    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-data\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.487679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487540    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-socket\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.487679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487586    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-kube-api-access-tmqmm\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-kube-api-access-tmqmm\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.487679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487616    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-certs\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.487679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487648    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-token\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-token\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487684    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-credential-socket\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487719    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-podinfo\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487771    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istiod-ca-cert\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487807    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-podinfo\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487817    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-data" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487832    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-socket" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487857    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-data\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.487914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487887    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-socket\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487919    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istiod-ca-cert\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487947    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-kube-api-access-2gljh\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-kube-api-access-2gljh\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.487979    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-certs\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488011    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-credential-socket\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488036    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-credential-socket" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488043    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-envoy\") pod \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\" (UID: \"401ed0a9-2ba8-403e-82b0-8d73081b8b5e\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488122    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istiod-ca-cert" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488128    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-envoy\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488166    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-token\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-token\") pod \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\" (UID: \"486a72db-72e2-40eb-9d6a-77ca411b7fd7\") "
Oct 30 19:15:49.488317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488231    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-certs" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.488861 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488448    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-data" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.488861 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488506    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.488861 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488525    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.488861 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488539    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.488861 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488553    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.488861 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488571    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.488861 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488802    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-certs" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.489237 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.488985    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-socket" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.489237 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.489010    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-credential-socket" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.489368 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.489273    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istiod-ca-cert" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:15:49.490883 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.490860    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-token" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:49.491019 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.490938    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-kube-api-access-tmqmm" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "kube-api-access-tmqmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:49.491019 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.490965    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-podinfo" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:49.491453 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.491430    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-token" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:49.491598 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.491578    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-envoy" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.492488 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.492467    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-podinfo" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:15:49.492585 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.492568    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-envoy" pod "486a72db-72e2-40eb-9d6a-77ca411b7fd7" (UID: "486a72db-72e2-40eb-9d6a-77ca411b7fd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:15:49.492725 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.492704    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-kube-api-access-2gljh" pod "401ed0a9-2ba8-403e-82b0-8d73081b8b5e" (UID: "401ed0a9-2ba8-403e-82b0-8d73081b8b5e"). InnerVolumeSpecName "kube-api-access-2gljh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:15:49.540909 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.540849    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" event={"ID":"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0","Type":"ContainerStarted","Data":"bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d"}
Oct 30 19:15:49.541102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.541002    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" event={"ID":"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0","Type":"ContainerStarted","Data":"b0c3402a226e943fa4265c7f9d06d74ef4fb57fa03b36a57fbd60521af5b148e"}
Oct 30 19:15:49.542276 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.542249    5025 generic.go:356] "Generic (PLEG): container finished" podID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerID="4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7" exitCode=0
Oct 30 19:15:49.542434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.542302    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"
Oct 30 19:15:49.542434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.542350    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" event={"ID":"486a72db-72e2-40eb-9d6a-77ca411b7fd7","Type":"ContainerDied","Data":"4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7"}
Oct 30 19:15:49.542434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.542406    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b" event={"ID":"486a72db-72e2-40eb-9d6a-77ca411b7fd7","Type":"ContainerDied","Data":"34be035f277e93f845025568d1cfc46c2389d92abdee27c13777a7ad68d85701"}
Oct 30 19:15:49.542434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.542433    5025 scope.go:122] "RemoveContainer" containerID="4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7"
Oct 30 19:15:49.544171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.544145    5025 generic.go:356] "Generic (PLEG): container finished" podID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerID="22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80" exitCode=0
Oct 30 19:15:49.544282 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.544188    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"
Oct 30 19:15:49.544282 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.544220    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" event={"ID":"401ed0a9-2ba8-403e-82b0-8d73081b8b5e","Type":"ContainerDied","Data":"22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80"}
Oct 30 19:15:49.544282 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.544262    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl" event={"ID":"401ed0a9-2ba8-403e-82b0-8d73081b8b5e","Type":"ContainerDied","Data":"0f830025c74aecc324aca1b32cfb812b38c2ed5fc4dc6110c2e080d8ec27aab5"}
Oct 30 19:15:49.553450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.553418    5025 scope.go:122] "RemoveContainer" containerID="4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7"
Oct 30 19:15:49.553725 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:15:49.553705    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7\": container with ID starting with 4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7 not found: ID does not exist" containerID="4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7"
Oct 30 19:15:49.553816 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.553738    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7"} err="failed to get container status \"4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7\": rpc error: code = NotFound desc = could not find container \"4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7\": container with ID starting with 4cd832c32d8f13de5d67dacfa49515b11114b7fe442fb5cff7f81d82f9c25cc7 not found: ID does not exist"
Oct 30 19:15:49.553816 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.553761    5025 scope.go:122] "RemoveContainer" containerID="22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80"
Oct 30 19:15:49.562086 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.562063    5025 scope.go:122] "RemoveContainer" containerID="47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33"
Oct 30 19:15:49.570144 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.570126    5025 scope.go:122] "RemoveContainer" containerID="22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80"
Oct 30 19:15:49.570472 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:15:49.570453    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80\": container with ID starting with 22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80 not found: ID does not exist" containerID="22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80"
Oct 30 19:15:49.570527 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.570479    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80"} err="failed to get container status \"22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80\": rpc error: code = NotFound desc = could not find container \"22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80\": container with ID starting with 22c9f799dc7f0f90459d49e9979d19cf566226c587a01af1835f4d91b3b08d80 not found: ID does not exist"
Oct 30 19:15:49.570527 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.570497    5025 scope.go:122] "RemoveContainer" containerID="47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33"
Oct 30 19:15:49.570746 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:15:49.570731    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33\": container with ID starting with 47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33 not found: ID does not exist" containerID="47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33"
Oct 30 19:15:49.570787 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.570751    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33"} err="failed to get container status \"47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33\": rpc error: code = NotFound desc = could not find container \"47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33\": container with ID starting with 47993512e1759c0ec666d5ead7916710e254ab6668b3c051c51e2bc867a05d33 not found: ID does not exist"
Oct 30 19:15:49.589156 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589128    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589156 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589156    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589173    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmqmm\" (UniqueName: \"kubernetes.io/projected/486a72db-72e2-40eb-9d6a-77ca411b7fd7-kube-api-access-tmqmm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589188    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589200    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589267    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589281    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589294    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589309    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/486a72db-72e2-40eb-9d6a-77ca411b7fd7-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589346    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gljh\" (UniqueName: \"kubernetes.io/projected/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-kube-api-access-2gljh\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589362    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589713 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589375    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/486a72db-72e2-40eb-9d6a-77ca411b7fd7-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.589713 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.589388    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/401ed0a9-2ba8-403e-82b0-8d73081b8b5e-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:15:49.700441 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.700363    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"]
Oct 30 19:15:49.708853 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.708821    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-9xt5b"]
Oct 30 19:15:49.770875 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.770839    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"]
Oct 30 19:15:49.777470 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:15:49.777441    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode79e3736_b546_400c_8137_2d07449be330.slice/crio-bea7b24b1d4cb2ed6cf939a69c9423faacee33c430c4d3b8ef6c8ca698015300 WatchSource:0}: Error finding container bea7b24b1d4cb2ed6cf939a69c9423faacee33c430c4d3b8ef6c8ca698015300: Status 404 returned error can't find the container with id bea7b24b1d4cb2ed6cf939a69c9423faacee33c430c4d3b8ef6c8ca698015300
Oct 30 19:15:49.816314 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.816285    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"]
Oct 30 19:15:49.823677 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:49.823653    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-r79wl"]
Oct 30 19:15:50.057121 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:50.057036    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" path="/var/lib/kubelet/pods/401ed0a9-2ba8-403e-82b0-8d73081b8b5e/volumes"
Oct 30 19:15:50.057510 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:50.057493    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" path="/var/lib/kubelet/pods/486a72db-72e2-40eb-9d6a-77ca411b7fd7/volumes"
Oct 30 19:15:50.552494 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:50.552456    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" event={"ID":"e79e3736-b546-400c-8137-2d07449be330","Type":"ContainerStarted","Data":"3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8"}
Oct 30 19:15:50.552690 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:50.552674    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:15:50.552746 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:50.552719    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" event={"ID":"e79e3736-b546-400c-8137-2d07449be330","Type":"ContainerStarted","Data":"bea7b24b1d4cb2ed6cf939a69c9423faacee33c430c4d3b8ef6c8ca698015300"}
Oct 30 19:15:50.672450 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:50.672390    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" podStartSLOduration=1.67236959 podStartE2EDuration="1.67236959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:50.670547903 +0000 UTC m=+2332.751134137" watchObservedRunningTime="2026-10-30 19:15:50.67236959 +0000 UTC m=+2332.752955797"
Oct 30 19:15:53.571699 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:53.571663    5025 generic.go:356] "Generic (PLEG): container finished" podID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerID="bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d" exitCode=0
Oct 30 19:15:53.572155 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:53.571742    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" event={"ID":"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0","Type":"ContainerDied","Data":"bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d"}
Oct 30 19:15:54.578580 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:54.578545    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" event={"ID":"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0","Type":"ContainerStarted","Data":"25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a"}
Oct 30 19:15:54.790434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:54.790381    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podStartSLOduration=3.349922436 podStartE2EDuration="6.790367168s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.440444732s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:15:54.741269364 +0000 UTC m=+2336.821855570" watchObservedRunningTime="2026-10-30 19:15:54.790367168 +0000 UTC m=+2336.870953374"
Oct 30 19:15:54.949974 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:54.949939    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:58.082088 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:15:58.082053    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:15:58.950695 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:58.950658    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:15:58.952525 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:15:58.952490    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:16:08.950374 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:08.950301    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:16:18.951150 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:18.951098    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:16:21.562747 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:21.562712    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:16:28.082654 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:16:28.082613    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:16:28.950706 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:28.950664    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:16:38.951224 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:38.951111    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:16:48.950788 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:48.950750    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:16:58.082519 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:16:58.082481    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:16:58.389199 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:58.389170    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:16:58.392905 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:58.392882    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:16:58.950461 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:16:58.950413    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:17:08.960165 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:08.960135    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:17:08.967717 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:08.967683    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:17:28.082372 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:17:28.082311    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:17:37.700851 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:37.700818    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"]
Oct 30 19:17:37.702381 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:37.702359    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"]
Oct 30 19:17:41.562671 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:41.562633    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:51.562491 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:51.562454    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:17:52.722054 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:52.722005    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main" containerID="cri-o://3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8" gracePeriod=45
Oct 30 19:17:52.723831 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:52.723799    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" containerID="cri-o://25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a" gracePeriod=45
Oct 30 19:17:52.961038 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:52.960997    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/health\": dial tcp 10.132.0.75:8000: connect: connection refused"
Oct 30 19:17:53.376897 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.376870    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:17:53.414077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.413997    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-home\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-home\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414035    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-cabundle-cert\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414097    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-dshm\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414123    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tls-certs\" (UniqueName: \"kubernetes.io/secret/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tls-certs\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414272    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kserve-provision-location\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414350    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-model-cache\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414385    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tmp-dir\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414416    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kube-api-access-pxr6x\" (UniqueName: \"kubernetes.io/projected/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kube-api-access-pxr6x\") pod \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\" (UID: \"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0\") "
Oct 30 19:17:53.414649 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414462    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-home" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:53.414649 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414585    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-cabundle-cert" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:17:53.414785 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414761    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-model-cache" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:53.414843 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414804    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.414843 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.414824    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.416853 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.416821    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kube-api-access-pxr6x" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "kube-api-access-pxr6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:17:53.416972 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.416901    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-dshm" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:53.416972 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.416923    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tls-certs" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:17:53.431795 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.431763    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tmp-dir" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:53.437961 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.437939    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kserve-provision-location" pod "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" (UID: "4e290002-fa1d-43fa-b14a-2ab7c1f36ef0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:17:53.516266 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.516230    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.516266 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.516260    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.516266 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.516269    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.516498 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.516281    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.516498 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.516295    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:53.516498 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:53.516305    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxr6x\" (UniqueName: \"kubernetes.io/projected/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0-kube-api-access-pxr6x\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:17:54.156961 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.156921    5025 generic.go:356] "Generic (PLEG): container finished" podID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerID="25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a" exitCode=0
Oct 30 19:17:54.157380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.156983    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"
Oct 30 19:17:54.157380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.157001    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" event={"ID":"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0","Type":"ContainerDied","Data":"25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a"}
Oct 30 19:17:54.157380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.157061    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7" event={"ID":"4e290002-fa1d-43fa-b14a-2ab7c1f36ef0","Type":"ContainerDied","Data":"b0c3402a226e943fa4265c7f9d06d74ef4fb57fa03b36a57fbd60521af5b148e"}
Oct 30 19:17:54.157380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.157086    5025 scope.go:122] "RemoveContainer" containerID="25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a"
Oct 30 19:17:54.165628 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.165610    5025 scope.go:122] "RemoveContainer" containerID="bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d"
Oct 30 19:17:54.173722 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.173697    5025 scope.go:122] "RemoveContainer" containerID="25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a"
Oct 30 19:17:54.173974 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:17:54.173955    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a\": container with ID starting with 25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a not found: ID does not exist" containerID="25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a"
Oct 30 19:17:54.174022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.173984    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a"} err="failed to get container status \"25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a\": rpc error: code = NotFound desc = could not find container \"25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a\": container with ID starting with 25bb54e57b909571e27e63c02360d193dc0c51ed1f2eff8b7a21935435d35b2a not found: ID does not exist"
Oct 30 19:17:54.174022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.174003    5025 scope.go:122] "RemoveContainer" containerID="bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d"
Oct 30 19:17:54.174237 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:17:54.174218    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d\": container with ID starting with bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d not found: ID does not exist" containerID="bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d"
Oct 30 19:17:54.174286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.174241    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d"} err="failed to get container status \"bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d\": rpc error: code = NotFound desc = could not find container \"bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d\": container with ID starting with bdc62b929663b777fae2fb014385322ed444f83f92b0535729d712cc9cc2ab8d not found: ID does not exist"
Oct 30 19:17:54.366105 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.366026    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"]
Oct 30 19:17:54.413332 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:54.413288    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-dd6f4dd47-9rwt7"]
Oct 30 19:17:56.056227 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:17:56.056193    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" path="/var/lib/kubelet/pods/4e290002-fa1d-43fa-b14a-2ab7c1f36ef0/volumes"
Oct 30 19:17:58.081603 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:17:58.081570    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:18:01.563004 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:01.562964    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:01.563509 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:01.563082    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:18:11.562727 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:11.562642    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:21.562294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:21.562251    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:18:22.980801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:22.980777    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:18:23.078102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.078024    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e79e3736-b546-400c-8137-2d07449be330-tls-certs\" (UniqueName: \"kubernetes.io/secret/e79e3736-b546-400c-8137-2d07449be330-tls-certs\") pod \"e79e3736-b546-400c-8137-2d07449be330\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") "
Oct 30 19:18:23.078102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.078099    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e79e3736-b546-400c-8137-2d07449be330-kube-api-access-tz5bm\" (UniqueName: \"kubernetes.io/projected/e79e3736-b546-400c-8137-2d07449be330-kube-api-access-tz5bm\") pod \"e79e3736-b546-400c-8137-2d07449be330\" (UID: \"e79e3736-b546-400c-8137-2d07449be330\") "
Oct 30 19:18:23.080218 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.080186    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e79e3736-b546-400c-8137-2d07449be330-tls-certs" pod "e79e3736-b546-400c-8137-2d07449be330" (UID: "e79e3736-b546-400c-8137-2d07449be330"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:18:23.080218 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.080190    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e79e3736-b546-400c-8137-2d07449be330-kube-api-access-tz5bm" pod "e79e3736-b546-400c-8137-2d07449be330" (UID: "e79e3736-b546-400c-8137-2d07449be330"). InnerVolumeSpecName "kube-api-access-tz5bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:18:23.178693 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.178661    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e79e3736-b546-400c-8137-2d07449be330-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:23.178693 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.178686    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz5bm\" (UniqueName: \"kubernetes.io/projected/e79e3736-b546-400c-8137-2d07449be330-kube-api-access-tz5bm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:18:23.284370 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.284313    5025 generic.go:356] "Generic (PLEG): container finished" podID="e79e3736-b546-400c-8137-2d07449be330" containerID="3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8" exitCode=0
Oct 30 19:18:23.284370 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.284372    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" event={"ID":"e79e3736-b546-400c-8137-2d07449be330","Type":"ContainerDied","Data":"3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8"}
Oct 30 19:18:23.284612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.284394    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"
Oct 30 19:18:23.284612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.284410    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98" event={"ID":"e79e3736-b546-400c-8137-2d07449be330","Type":"ContainerDied","Data":"bea7b24b1d4cb2ed6cf939a69c9423faacee33c430c4d3b8ef6c8ca698015300"}
Oct 30 19:18:23.284612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.284434    5025 scope.go:122] "RemoveContainer" containerID="3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8"
Oct 30 19:18:23.294087 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.294063    5025 scope.go:122] "RemoveContainer" containerID="3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8"
Oct 30 19:18:23.296523 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:18:23.296502    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8\": container with ID starting with 3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8 not found: ID does not exist" containerID="3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8"
Oct 30 19:18:23.296619 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.296533    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8"} err="failed to get container status \"3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8\": rpc error: code = NotFound desc = could not find container \"3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8\": container with ID starting with 3750620b84cd8003aa010c2c753fed416e46c7957c4f5ba2f7b3fc8c3d4679d8 not found: ID does not exist"
Oct 30 19:18:23.447941 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.447907    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"]
Oct 30 19:18:23.453683 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:23.453656    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schen4f98"]
Oct 30 19:18:24.056933 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:24.056901    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e79e3736-b546-400c-8137-2d07449be330" path="/var/lib/kubelet/pods/e79e3736-b546-400c-8137-2d07449be330/volumes"
Oct 30 19:18:28.089427 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:18:28.089233    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:18:43.609546 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.609516    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"]
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610031    5025 cpu_manager.go:414] "containerMap: removing container" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610049    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610074    5025 cpu_manager.go:414] "containerMap: removing container" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="storage-initializer"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610082    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="storage-initializer"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610098    5025 cpu_manager.go:414] "containerMap: removing container" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610107    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610119    5025 cpu_manager.go:414] "containerMap: removing container" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610126    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610146    5025 cpu_manager.go:414] "containerMap: removing container" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy"
Oct 30 19:18:43.610158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610153    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy"
Oct 30 19:18:43.610674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610184    5025 cpu_manager.go:414] "containerMap: removing container" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main"
Oct 30 19:18:43.610674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610192    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main"
Oct 30 19:18:43.610674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610283    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="486a72db-72e2-40eb-9d6a-77ca411b7fd7" containerName="istio-proxy"
Oct 30 19:18:43.610674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610301    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e290002-fa1d-43fa-b14a-2ab7c1f36ef0" containerName="main"
Oct 30 19:18:43.610674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610311    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy"
Oct 30 19:18:43.610674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610340    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="e79e3736-b546-400c-8137-2d07449be330" containerName="main"
Oct 30 19:18:43.610674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.610350    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="401ed0a9-2ba8-403e-82b0-8d73081b8b5e" containerName="istio-proxy"
Oct 30 19:18:43.613685 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.613664    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641025 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.640998    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641130 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.641031    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641130 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.641052    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d479d2f-fc67-479c-9f1b-4a8977155740-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.641165    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.641199    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d479d2f-fc67-479c-9f1b-4a8977155740-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641272 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.641262    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4tc6\" (UniqueName: \"kubernetes.io/projected/0d479d2f-fc67-479c-9f1b-4a8977155740-kube-api-access-d4tc6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641306 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.641290    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.641368 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.641358    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.646900 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.646875    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"]
Oct 30 19:18:43.692837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.692810    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:18:43.692837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.692813    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Oct 30 19:18:43.692988 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.692840    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Oct 30 19:18:43.692988 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.692943    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Oct 30 19:18:43.693072 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.692994    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-vjzr7\""
Oct 30 19:18:43.693072 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.692999    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Oct 30 19:18:43.742467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742443    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.742582 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742474    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d479d2f-fc67-479c-9f1b-4a8977155740-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.742582 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742496    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.742582 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742525    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d479d2f-fc67-479c-9f1b-4a8977155740-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.742582 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742556    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4tc6\" (UniqueName: \"kubernetes.io/projected/0d479d2f-fc67-479c-9f1b-4a8977155740-kube-api-access-d4tc6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.742849 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742586    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.742849 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742630    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.742849 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742678    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.743010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742911    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.743010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.742972    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.743216 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.743190    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.743349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.743232    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.743349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.743246    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d479d2f-fc67-479c-9f1b-4a8977155740-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.745101 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.745081    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d479d2f-fc67-479c-9f1b-4a8977155740-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.745182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.745084    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.798775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.798752    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4tc6\" (UniqueName: \"kubernetes.io/projected/0d479d2f-fc67-479c-9f1b-4a8977155740-kube-api-access-d4tc6\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:43.928893 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:43.928854    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:44.026329 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.026295    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"]
Oct 30 19:18:44.030016 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.029996    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.084178 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.084144    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-k8cvh\""
Oct 30 19:18:44.085518 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.085491    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"]
Oct 30 19:18:44.146253 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.146222    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tw2f\" (UniqueName: \"kubernetes.io/projected/317ff98a-938f-48e6-8f63-31a1b9b762d0-kube-api-access-9tw2f\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.146414 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.146305    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317ff98a-938f-48e6-8f63-31a1b9b762d0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.247756 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.247661    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tw2f\" (UniqueName: \"kubernetes.io/projected/317ff98a-938f-48e6-8f63-31a1b9b762d0-kube-api-access-9tw2f\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.247756 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.247752    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317ff98a-938f-48e6-8f63-31a1b9b762d0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.250157 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.250128    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317ff98a-938f-48e6-8f63-31a1b9b762d0-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.303884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.303853    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tw2f\" (UniqueName: \"kubernetes.io/projected/317ff98a-938f-48e6-8f63-31a1b9b762d0-kube-api-access-9tw2f\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.343399 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.343375    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:44.352690 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.352669    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"]
Oct 30 19:18:44.357754 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:18:44.357730    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d479d2f_fc67_479c_9f1b_4a8977155740.slice/crio-e38a74f13ca4a12116ae6b1c3af184c8b8221684329c9f6244ce55c892da8854 WatchSource:0}: Error finding container e38a74f13ca4a12116ae6b1c3af184c8b8221684329c9f6244ce55c892da8854: Status 404 returned error can't find the container with id e38a74f13ca4a12116ae6b1c3af184c8b8221684329c9f6244ce55c892da8854
Oct 30 19:18:44.387443 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.387416    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" event={"ID":"0d479d2f-fc67-479c-9f1b-4a8977155740","Type":"ContainerStarted","Data":"e38a74f13ca4a12116ae6b1c3af184c8b8221684329c9f6244ce55c892da8854"}
Oct 30 19:18:44.712763 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:44.712727    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"]
Oct 30 19:18:44.718363 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:18:44.718338    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317ff98a_938f_48e6_8f63_31a1b9b762d0.slice/crio-ad34c51c91737d1f5677337fe3ef1432ec8fddd56f76199ed8dd13e283bd6ea9 WatchSource:0}: Error finding container ad34c51c91737d1f5677337fe3ef1432ec8fddd56f76199ed8dd13e283bd6ea9: Status 404 returned error can't find the container with id ad34c51c91737d1f5677337fe3ef1432ec8fddd56f76199ed8dd13e283bd6ea9
Oct 30 19:18:45.393695 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:45.393658    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" event={"ID":"0d479d2f-fc67-479c-9f1b-4a8977155740","Type":"ContainerStarted","Data":"6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363"}
Oct 30 19:18:45.395201 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:45.395174    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" event={"ID":"317ff98a-938f-48e6-8f63-31a1b9b762d0","Type":"ContainerStarted","Data":"f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7"}
Oct 30 19:18:45.395355 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:45.395216    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" event={"ID":"317ff98a-938f-48e6-8f63-31a1b9b762d0","Type":"ContainerStarted","Data":"ad34c51c91737d1f5677337fe3ef1432ec8fddd56f76199ed8dd13e283bd6ea9"}
Oct 30 19:18:45.395355 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:45.395307    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:18:45.625939 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:45.625886    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" podStartSLOduration=2.625873334 podStartE2EDuration="2.625873334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:45.624387224 +0000 UTC m=+2507.704973431" watchObservedRunningTime="2026-10-30 19:18:45.625873334 +0000 UTC m=+2507.706459542"
Oct 30 19:18:48.410033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:48.410000    5025 generic.go:356] "Generic (PLEG): container finished" podID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerID="6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363" exitCode=0
Oct 30 19:18:48.410454 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:48.410075    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" event={"ID":"0d479d2f-fc67-479c-9f1b-4a8977155740","Type":"ContainerDied","Data":"6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363"}
Oct 30 19:18:49.417303 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:49.417270    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" event={"ID":"0d479d2f-fc67-479c-9f1b-4a8977155740","Type":"ContainerStarted","Data":"884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3"}
Oct 30 19:18:49.631724 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:49.631672    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podStartSLOduration=3.312719661 podStartE2EDuration="6.631657419s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.318937758s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:18:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:18:49.583940422 +0000 UTC m=+2511.664526644" watchObservedRunningTime="2026-10-30 19:18:49.631657419 +0000 UTC m=+2511.712243625"
Oct 30 19:18:49.929309 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:49.929278    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:53.929502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:53.929462    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:18:53.930863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:18:53.930836    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:18:58.082462 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:18:58.082425    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:19:00.656650 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.656613    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"]
Oct 30 19:19:00.660574 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.660550    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702397 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702365    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702525 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702426    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702525 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702506    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702599 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702547    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702599 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702563    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702599 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702585    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702605    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702655    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4091c6a4-fecc-45a4-92ad-ce82248ea675-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.702847 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.702825    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pngz\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-kube-api-access-8pngz\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804421 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804378    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pngz\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-kube-api-access-8pngz\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804606 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804471    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804606 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804518    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804606 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804572    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804616    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804645    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804697    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.804988 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804965    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.805054 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.804998    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.805054 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.805024    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.805167 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.805077    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4091c6a4-fecc-45a4-92ad-ce82248ea675-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.805338 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.805281    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.805467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.805364    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.805851 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.805826    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4091c6a4-fecc-45a4-92ad-ce82248ea675-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.806971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.806949    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.807597 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.807577    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.856811 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.856779    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"]
Oct 30 19:19:00.862874 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.862840    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.863043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.862980    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pngz\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-kube-api-access-8pngz\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-nhvlw\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:00.974050 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:00.973970    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:01.397225 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.397181    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"]
Oct 30 19:19:01.402773 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:19:01.402738    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4091c6a4_fecc_45a4_92ad_ce82248ea675.slice/crio-0877565ddf6eeb749e16518b97ecb419acf49b7786e1db0e3d7259a738997ea7 WatchSource:0}: Error finding container 0877565ddf6eeb749e16518b97ecb419acf49b7786e1db0e3d7259a738997ea7: Status 404 returned error can't find the container with id 0877565ddf6eeb749e16518b97ecb419acf49b7786e1db0e3d7259a738997ea7
Oct 30 19:19:01.404989 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.404953    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:01.405109 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.405025    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:01.405109 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.405054    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:01.486981 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.486936    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" event={"ID":"4091c6a4-fecc-45a4-92ad-ce82248ea675","Type":"ContainerStarted","Data":"8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9"}
Oct 30 19:19:01.487164 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.487049    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" event={"ID":"4091c6a4-fecc-45a4-92ad-ce82248ea675","Type":"ContainerStarted","Data":"0877565ddf6eeb749e16518b97ecb419acf49b7786e1db0e3d7259a738997ea7"}
Oct 30 19:19:01.604648 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.604584    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podStartSLOduration=1.604564607 podStartE2EDuration="1.604564607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:01.603142872 +0000 UTC m=+2523.683729098" watchObservedRunningTime="2026-10-30 19:19:01.604564607 +0000 UTC m=+2523.685150816"
Oct 30 19:19:01.977481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:01.977437    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:02.982351 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:02.980415    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:19:02.982351 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:02.980477    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:19:03.929867 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:03.929829    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:19:03.975831 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:03.975788    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:19:03.976125 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:03.976094    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:19:04.977345 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:04.975726    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:19:04.977345 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:04.975814    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:19:05.975343 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:05.974855    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:19:05.975343 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:05.974918    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:19:06.979348 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:06.976676    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body=
Oct 30 19:19:06.979348 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:06.976750    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused"
Oct 30 19:19:07.978943 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:07.978912    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:07.979366 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:07.979340    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:07.979986 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:07.979964    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:19:13.929541 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:13.929492    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:19:15.774901 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.774863    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"]
Oct 30 19:19:15.781597 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.781569    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857420 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857369    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d6a566-5379-474e-8776-3cad924206a2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857595 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857541    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857595 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857570    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857691 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857592    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h49ht\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-kube-api-access-h49ht\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857691 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857670    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857797 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857693    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d6a566-5379-474e-8776-3cad924206a2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857854 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857802    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857854 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857841    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.857964 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.857911    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959129 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959095    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959129 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959130    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959394 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959151    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h49ht\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-kube-api-access-h49ht\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959394 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959174    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959394 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959194    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d6a566-5379-474e-8776-3cad924206a2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959394 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959308    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959394 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959383    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959654 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959417    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959654 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959519    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d6a566-5379-474e-8776-3cad924206a2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959653    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959723    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959877 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959822    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.959991 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959964    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.960041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.959997    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d6a566-5379-474e-8776-3cad924206a2-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.962026 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.961996    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.962158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.962140    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d6a566-5379-474e-8776-3cad924206a2-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:15.983371 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:15.983312    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"]
Oct 30 19:19:16.016961 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.016924    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:16.017310 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.017287    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h49ht\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-kube-api-access-h49ht\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-92qxs\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:16.095033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.094952    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:16.401518 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.401488    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:19:16.522882 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.522840    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"]
Oct 30 19:19:16.528462 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:19:16.528436    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d6a566_5379_474e_8776_3cad924206a2.slice/crio-41df6993c10e415f6a82af20d839c17fed4628e9fbde540bd548d0ff482ac4b2 WatchSource:0}: Error finding container 41df6993c10e415f6a82af20d839c17fed4628e9fbde540bd548d0ff482ac4b2: Status 404 returned error can't find the container with id 41df6993c10e415f6a82af20d839c17fed4628e9fbde540bd548d0ff482ac4b2
Oct 30 19:19:16.530713 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.530673    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:16.530801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.530760    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:16.530842 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.530806    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:19:16.560314 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:16.560270    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" event={"ID":"c4d6a566-5379-474e-8776-3cad924206a2","Type":"ContainerStarted","Data":"41df6993c10e415f6a82af20d839c17fed4628e9fbde540bd548d0ff482ac4b2"}
Oct 30 19:19:17.588495 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:17.585500    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" event={"ID":"c4d6a566-5379-474e-8776-3cad924206a2","Type":"ContainerStarted","Data":"e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43"}
Oct 30 19:19:17.750022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:17.749961    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podStartSLOduration=2.7499463779999997 podStartE2EDuration="2.749946378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:19:17.748770881 +0000 UTC m=+2539.829357109" watchObservedRunningTime="2026-10-30 19:19:17.749946378 +0000 UTC m=+2539.830532585"
Oct 30 19:19:18.095095 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:18.095062    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:18.096589 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:18.096562    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-92qxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:19:18.096722 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:18.096609    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:19:19.096277 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:19.096230    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-92qxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:19:19.096785 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:19.096299    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:19:20.096341 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:20.096296    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-92qxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:19:20.096818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:20.096383    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:19:21.096302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:21.096257    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-92qxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:19:21.096519 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:21.096338    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:19:22.095828 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:22.095791    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-92qxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 30 19:19:22.096010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:22.095843    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 30 19:19:23.100103 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:23.100072    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:23.100572 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:23.100354    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:23.101000 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:23.100984    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:19:23.930043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:23.929999    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:19:28.083556 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:19:28.083523    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:19:33.929921 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:33.929884    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:19:43.929312 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:43.929220    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:19:53.929582 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:19:53.929529    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:19:58.082382 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:19:58.082343    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:20:03.939260 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:03.939230    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:20:03.946762 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:03.946738    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:20:10.661869 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:10.661837    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"]
Oct 30 19:20:10.666269 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:10.666231    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"]
Oct 30 19:20:16.401337 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:16.401281    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:25.688786 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:25.688720    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main" containerID="cri-o://f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7" gracePeriod=45
Oct 30 19:20:25.692953 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:25.692898    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" containerID="cri-o://884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3" gracePeriod=45
Oct 30 19:20:25.940294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:25.940204    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Oct 30 19:20:26.345794 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.345769    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:20:26.401921 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.401885    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:26.424172 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424142    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-home\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-home\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424193    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-model-cache\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424238    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d479d2f-fc67-479c-9f1b-4a8977155740-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d479d2f-fc67-479c-9f1b-4a8977155740-tls-certs\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424269    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d479d2f-fc67-479c-9f1b-4a8977155740-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d479d2f-fc67-479c-9f1b-4a8977155740-cabundle-cert\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424504 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424389    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d479d2f-fc67-479c-9f1b-4a8977155740-kube-api-access-d4tc6\" (UniqueName: \"kubernetes.io/projected/0d479d2f-fc67-479c-9f1b-4a8977155740-kube-api-access-d4tc6\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424504 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424420    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-dshm\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424504 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424448    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-kserve-provision-location\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424504 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424478    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-tmp-dir\") pod \"0d479d2f-fc67-479c-9f1b-4a8977155740\" (UID: \"0d479d2f-fc67-479c-9f1b-4a8977155740\") "
Oct 30 19:20:26.424703 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424567    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-model-cache" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:26.424703 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424638    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-home" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:26.424703 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424684    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d479d2f-fc67-479c-9f1b-4a8977155740-cabundle-cert" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:20:26.424937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424915    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.425001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424943    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.425001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424953    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d479d2f-fc67-479c-9f1b-4a8977155740-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.425001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.424914    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-tmp-dir" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:26.426655 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.426629    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d479d2f-fc67-479c-9f1b-4a8977155740-tls-certs" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:26.427001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.426980    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-dshm" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:26.427117 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.427097    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d479d2f-fc67-479c-9f1b-4a8977155740-kube-api-access-d4tc6" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "kube-api-access-d4tc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:26.447437 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.447416    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-kserve-provision-location" pod "0d479d2f-fc67-479c-9f1b-4a8977155740" (UID: "0d479d2f-fc67-479c-9f1b-4a8977155740"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:20:26.525863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.525789    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d479d2f-fc67-479c-9f1b-4a8977155740-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.525863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.525817    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4tc6\" (UniqueName: \"kubernetes.io/projected/0d479d2f-fc67-479c-9f1b-4a8977155740-kube-api-access-d4tc6\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.525863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.525827    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.525863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.525837    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.525863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.525846    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d479d2f-fc67-479c-9f1b-4a8977155740-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:26.929179 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.929146    5025 generic.go:356] "Generic (PLEG): container finished" podID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerID="884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3" exitCode=0
Oct 30 19:20:26.929696 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.929202    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" event={"ID":"0d479d2f-fc67-479c-9f1b-4a8977155740","Type":"ContainerDied","Data":"884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3"}
Oct 30 19:20:26.929696 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.929206    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"
Oct 30 19:20:26.929696 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.929242    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4" event={"ID":"0d479d2f-fc67-479c-9f1b-4a8977155740","Type":"ContainerDied","Data":"e38a74f13ca4a12116ae6b1c3af184c8b8221684329c9f6244ce55c892da8854"}
Oct 30 19:20:26.929696 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.929264    5025 scope.go:122] "RemoveContainer" containerID="884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3"
Oct 30 19:20:26.938470 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.938451    5025 scope.go:122] "RemoveContainer" containerID="6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363"
Oct 30 19:20:26.947024 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.947004    5025 scope.go:122] "RemoveContainer" containerID="884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3"
Oct 30 19:20:26.947271 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:20:26.947249    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3\": container with ID starting with 884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3 not found: ID does not exist" containerID="884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3"
Oct 30 19:20:26.947371 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.947282    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3"} err="failed to get container status \"884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3\": rpc error: code = NotFound desc = could not find container \"884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3\": container with ID starting with 884260593ccf98a55104c702a80c4955d1098371de224260c48e6b1c5149eff3 not found: ID does not exist"
Oct 30 19:20:26.947371 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.947306    5025 scope.go:122] "RemoveContainer" containerID="6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363"
Oct 30 19:20:26.947578 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:20:26.947563    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363\": container with ID starting with 6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363 not found: ID does not exist" containerID="6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363"
Oct 30 19:20:26.947616 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:26.947585    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363"} err="failed to get container status \"6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363\": rpc error: code = NotFound desc = could not find container \"6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363\": container with ID starting with 6fee47fa9067570a67b0f902881cac51a1ebbe593387d4df4205af83fc697363 not found: ID does not exist"
Oct 30 19:20:27.091301 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:27.091263    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"]
Oct 30 19:20:27.138920 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:27.138891    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-86b88cb9c4-mjxp4"]
Oct 30 19:20:28.056472 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:28.056435    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" path="/var/lib/kubelet/pods/0d479d2f-fc67-479c-9f1b-4a8977155740/volumes"
Oct 30 19:20:28.082094 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:20:28.082064    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:20:36.401263 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:36.401219    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:36.401670 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:36.401396    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:20:46.401958 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:46.401917    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:20:55.946920 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:55.946894    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:20:56.066522 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.066442    5025 generic.go:356] "Generic (PLEG): container finished" podID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerID="f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7" exitCode=0
Oct 30 19:20:56.066522 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.066488    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"
Oct 30 19:20:56.066697 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.066527    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" event={"ID":"317ff98a-938f-48e6-8f63-31a1b9b762d0","Type":"ContainerDied","Data":"f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7"}
Oct 30 19:20:56.066697 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.066573    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f" event={"ID":"317ff98a-938f-48e6-8f63-31a1b9b762d0","Type":"ContainerDied","Data":"ad34c51c91737d1f5677337fe3ef1432ec8fddd56f76199ed8dd13e283bd6ea9"}
Oct 30 19:20:56.066697 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.066597    5025 scope.go:122] "RemoveContainer" containerID="f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7"
Oct 30 19:20:56.076900 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.076882    5025 scope.go:122] "RemoveContainer" containerID="f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7"
Oct 30 19:20:56.077204 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:20:56.077182    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7\": container with ID starting with f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7 not found: ID does not exist" containerID="f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7"
Oct 30 19:20:56.077276 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.077213    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7"} err="failed to get container status \"f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7\": rpc error: code = NotFound desc = could not find container \"f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7\": container with ID starting with f334fb2f871b00412d518475e11846258b09c273e91a5bf42db510d7fb9b01c7 not found: ID does not exist"
Oct 30 19:20:56.096039 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.096016    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/317ff98a-938f-48e6-8f63-31a1b9b762d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/317ff98a-938f-48e6-8f63-31a1b9b762d0-tls-certs\") pod \"317ff98a-938f-48e6-8f63-31a1b9b762d0\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") "
Oct 30 19:20:56.096175 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.096053    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317ff98a-938f-48e6-8f63-31a1b9b762d0-kube-api-access-9tw2f\" (UniqueName: \"kubernetes.io/projected/317ff98a-938f-48e6-8f63-31a1b9b762d0-kube-api-access-9tw2f\") pod \"317ff98a-938f-48e6-8f63-31a1b9b762d0\" (UID: \"317ff98a-938f-48e6-8f63-31a1b9b762d0\") "
Oct 30 19:20:56.098092 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.098068    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317ff98a-938f-48e6-8f63-31a1b9b762d0-kube-api-access-9tw2f" pod "317ff98a-938f-48e6-8f63-31a1b9b762d0" (UID: "317ff98a-938f-48e6-8f63-31a1b9b762d0"). InnerVolumeSpecName "kube-api-access-9tw2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:20:56.098178 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.098139    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/317ff98a-938f-48e6-8f63-31a1b9b762d0-tls-certs" pod "317ff98a-938f-48e6-8f63-31a1b9b762d0" (UID: "317ff98a-938f-48e6-8f63-31a1b9b762d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:20:56.197226 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.197191    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/317ff98a-938f-48e6-8f63-31a1b9b762d0-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:56.197226 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.197219    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tw2f\" (UniqueName: \"kubernetes.io/projected/317ff98a-938f-48e6-8f63-31a1b9b762d0-kube-api-access-9tw2f\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:20:56.490272 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.490233    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"]
Oct 30 19:20:56.495682 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:56.495656    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schept95f"]
Oct 30 19:20:58.056153 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:20:58.056115    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" path="/var/lib/kubelet/pods/317ff98a-938f-48e6-8f63-31a1b9b762d0/volumes"
Oct 30 19:20:58.083052 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:20:58.083021    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:21:07.355038 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.354960    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"]
Oct 30 19:21:07.355436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355363    5025 cpu_manager.go:414] "containerMap: removing container" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main"
Oct 30 19:21:07.355436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355375    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main"
Oct 30 19:21:07.355436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355389    5025 cpu_manager.go:414] "containerMap: removing container" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main"
Oct 30 19:21:07.355436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355394    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main"
Oct 30 19:21:07.355436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355425    5025 cpu_manager.go:414] "containerMap: removing container" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="storage-initializer"
Oct 30 19:21:07.355436 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355431    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="storage-initializer"
Oct 30 19:21:07.355633 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355496    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d479d2f-fc67-479c-9f1b-4a8977155740" containerName="main"
Oct 30 19:21:07.355633 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.355505    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="317ff98a-938f-48e6-8f63-31a1b9b762d0" containerName="main"
Oct 30 19:21:07.360208 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.360184    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387274 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387245    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kserve-provision-location\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387432 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387314    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-cabundle-cert\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387432 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387380    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jc66\" (UniqueName: \"kubernetes.io/projected/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kube-api-access-9jc66\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387546 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387442    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-dshm\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387546 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387462    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-home\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387546 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387482    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tls-certs\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387546 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387500    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-model-cache\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.387758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.387609    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tmp-dir\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.388029 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.388006    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"]
Oct 30 19:21:07.433645 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.433618    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Oct 30 19:21:07.433809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.433639    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:21:07.433809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.433668    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Oct 30 19:21:07.433809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.433680    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Oct 30 19:21:07.433928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.433704    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-dqkpq\""
Oct 30 19:21:07.433928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.433727    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Oct 30 19:21:07.488953 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.488923    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kserve-provision-location\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.488983    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-cabundle-cert\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.488999    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jc66\" (UniqueName: \"kubernetes.io/projected/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kube-api-access-9jc66\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489023    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-dshm\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489037    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-home\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489065    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tls-certs\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489201    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-model-cache\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489261    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tmp-dir\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489419    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kserve-provision-location\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489455    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-home\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489698 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489666    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-model-cache\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489698 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489688    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tmp-dir\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.489905 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.489883    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-cabundle-cert\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.491317 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.491288    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-dshm\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.491848 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.491832    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tls-certs\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.549498 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.549473    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jc66\" (UniqueName: \"kubernetes.io/projected/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kube-api-access-9jc66\") pod \"stop-feature-test-kserve-98446cffd-5pt65\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:07.675453 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:07.675418    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:08.097345 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:08.097247    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"]
Oct 30 19:21:08.103333 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:21:08.103294    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2ffd15_ea29_48c6_a7fd_a4dddf7b5116.slice/crio-0b46edd01dff3db0bfb4d4fb0588e7b7856f0045c68a7ede639d829f9390313d WatchSource:0}: Error finding container 0b46edd01dff3db0bfb4d4fb0588e7b7856f0045c68a7ede639d829f9390313d: Status 404 returned error can't find the container with id 0b46edd01dff3db0bfb4d4fb0588e7b7856f0045c68a7ede639d829f9390313d
Oct 30 19:21:08.123626 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:08.123599    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" event={"ID":"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116","Type":"ContainerStarted","Data":"0b46edd01dff3db0bfb4d4fb0588e7b7856f0045c68a7ede639d829f9390313d"}
Oct 30 19:21:09.129993 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:09.129951    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" event={"ID":"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116","Type":"ContainerStarted","Data":"d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49"}
Oct 30 19:21:11.140431 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:11.140395    5025 generic.go:356] "Generic (PLEG): container finished" podID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerID="d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49" exitCode=0
Oct 30 19:21:11.140813 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:11.140438    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" event={"ID":"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116","Type":"ContainerDied","Data":"d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49"}
Oct 30 19:21:12.147680 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:12.147640    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" event={"ID":"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116","Type":"ContainerStarted","Data":"602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911"}
Oct 30 19:21:12.359710 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:12.359641    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podStartSLOduration=3.1805928310000002 podStartE2EDuration="5.359621381s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.17902855s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:12.310833432 +0000 UTC m=+2654.391419641" watchObservedRunningTime="2026-10-30 19:21:12.359621381 +0000 UTC m=+2654.440207591"
Oct 30 19:21:12.676530 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:12.676499    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:16.325806 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.325765    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"]
Oct 30 19:21:16.330912 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.330889    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:16.370235 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.370202    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30e231f-b73b-4472-a7c0-75c2b75b33b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-wp668\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:16.370399 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.370248    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thzfr\" (UniqueName: \"kubernetes.io/projected/b30e231f-b73b-4472-a7c0-75c2b75b33b9-kube-api-access-thzfr\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-wp668\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:16.384661 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.384631    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"]
Oct 30 19:21:16.429700 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.429670    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-wwrlh\""
Oct 30 19:21:16.471489 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.471454    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30e231f-b73b-4472-a7c0-75c2b75b33b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-wp668\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:16.471674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.471510    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thzfr\" (UniqueName: \"kubernetes.io/projected/b30e231f-b73b-4472-a7c0-75c2b75b33b9-kube-api-access-thzfr\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-wp668\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:16.474892 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.474853    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30e231f-b73b-4472-a7c0-75c2b75b33b9-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-wp668\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:16.528184 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.528156    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thzfr\" (UniqueName: \"kubernetes.io/projected/b30e231f-b73b-4472-a7c0-75c2b75b33b9-kube-api-access-thzfr\") pod \"stop-feature-test-kserve-router-scheduler-6c5444664c-wp668\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:16.645177 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:16.645141    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:17.018693 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:17.018605    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"]
Oct 30 19:21:17.024091 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:21:17.024063    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb30e231f_b73b_4472_a7c0_75c2b75b33b9.slice/crio-de61fb6e8db1141590d521d3dd89f301be0d7a08db3c97a12ce2364abb5f0e51 WatchSource:0}: Error finding container de61fb6e8db1141590d521d3dd89f301be0d7a08db3c97a12ce2364abb5f0e51: Status 404 returned error can't find the container with id de61fb6e8db1141590d521d3dd89f301be0d7a08db3c97a12ce2364abb5f0e51
Oct 30 19:21:17.173851 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:17.173818    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" event={"ID":"b30e231f-b73b-4472-a7c0-75c2b75b33b9","Type":"ContainerStarted","Data":"af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932"}
Oct 30 19:21:17.174023 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:17.173871    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" event={"ID":"b30e231f-b73b-4472-a7c0-75c2b75b33b9","Type":"ContainerStarted","Data":"de61fb6e8db1141590d521d3dd89f301be0d7a08db3c97a12ce2364abb5f0e51"}
Oct 30 19:21:17.174023 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:17.173930    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:17.290212 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:17.290115    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" podStartSLOduration=1.29010199 podStartE2EDuration="1.29010199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:21:17.289141873 +0000 UTC m=+2659.369728082" watchObservedRunningTime="2026-10-30 19:21:17.29010199 +0000 UTC m=+2659.370688196"
Oct 30 19:21:17.676203 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:17.676169    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:21:17.677793 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:17.677763    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:21:27.676833 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:27.676783    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:21:28.082072 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:21:28.081983    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:21:37.676410 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:37.676362    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:21:47.676218 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:47.676179    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:21:48.180746 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:48.180717    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:21:57.676439 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:57.676400    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:21:58.082452 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:21:58.082366    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:21:58.420439 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:58.420407    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:21:58.426083 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:21:58.426058    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:22:07.676896 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:22:07.676845    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:22:17.676556 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:22:17.676514    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:22:27.676592 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:22:27.676538    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:22:28.082031 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:22:28.081937    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:22:37.676044 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:22:37.675959    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:22:47.676793 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:22:47.676745    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:22:57.676260 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:22:57.676210    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:22:58.094654 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:22:58.094557    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:23:07.685475 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:07.685445    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:23:07.693000 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:07.692974    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:23:14.374963 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:14.374927    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"]
Oct 30 19:23:14.492470 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:14.492432    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"]
Oct 30 19:23:18.180447 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:18.180407    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:24.697679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.697645    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"]
Oct 30 19:23:24.701924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.701903    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.745358 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.745301    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"]
Oct 30 19:23:24.859713 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.859677    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-model-cache\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.859886 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.859725    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-home\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.859886 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.859750    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3390d74-02de-49c0-92a6-3199b2f40395-tls-certs\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.859886 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.859810    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-kserve-provision-location\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.859886 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.859874    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-dshm\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.860027 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.859940    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3390d74-02de-49c0-92a6-3199b2f40395-cabundle-cert\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.860027 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.859977    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-tmp-dir\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.860096 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.860058    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9n5h\" (UniqueName: \"kubernetes.io/projected/e3390d74-02de-49c0-92a6-3199b2f40395-kube-api-access-f9n5h\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961446    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9n5h\" (UniqueName: \"kubernetes.io/projected/e3390d74-02de-49c0-92a6-3199b2f40395-kube-api-access-f9n5h\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961525    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-model-cache\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961739 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961551    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-home\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961739 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961570    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3390d74-02de-49c0-92a6-3199b2f40395-tls-certs\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961739 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961585    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-kserve-provision-location\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961739 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961680    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-dshm\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961950 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961739    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3390d74-02de-49c0-92a6-3199b2f40395-cabundle-cert\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.961950 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961783    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-tmp-dir\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.962052 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.961978    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-model-cache\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.962052 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.962011    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-home\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.962154 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.962069    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-kserve-provision-location\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.962222 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.962200    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-tmp-dir\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.962543 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.962524    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3390d74-02de-49c0-92a6-3199b2f40395-cabundle-cert\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.963960 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.963931    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-dshm\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:24.964417 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:24.964394    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3390d74-02de-49c0-92a6-3199b2f40395-tls-certs\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:25.018200 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:25.018161    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9n5h\" (UniqueName: \"kubernetes.io/projected/e3390d74-02de-49c0-92a6-3199b2f40395-kube-api-access-f9n5h\") pod \"stop-feature-test-kserve-98446cffd-6sfgp\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:25.317685 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:25.317592    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:25.811783 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:25.811741    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"]
Oct 30 19:23:25.817454 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:23:25.817424    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3390d74_02de_49c0_92a6_3199b2f40395.slice/crio-1a3b935bad88f93cfdd01a7dc5f0fab448ba093a199e8228c3c6e4f8d6585f3f WatchSource:0}: Error finding container 1a3b935bad88f93cfdd01a7dc5f0fab448ba093a199e8228c3c6e4f8d6585f3f: Status 404 returned error can't find the container with id 1a3b935bad88f93cfdd01a7dc5f0fab448ba093a199e8228c3c6e4f8d6585f3f
Oct 30 19:23:26.815770 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:26.815738    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" event={"ID":"e3390d74-02de-49c0-92a6-3199b2f40395","Type":"ContainerStarted","Data":"95076ca18b750ca2ca13659e828d1ba7d6e15660b91e67132d76680f88096f62"}
Oct 30 19:23:26.816162 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:26.815789    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" event={"ID":"e3390d74-02de-49c0-92a6-3199b2f40395","Type":"ContainerStarted","Data":"1a3b935bad88f93cfdd01a7dc5f0fab448ba093a199e8228c3c6e4f8d6585f3f"}
Oct 30 19:23:28.081806 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:23:28.081775    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:23:28.834351 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:28.834296    5025 generic.go:356] "Generic (PLEG): container finished" podID="e3390d74-02de-49c0-92a6-3199b2f40395" containerID="95076ca18b750ca2ca13659e828d1ba7d6e15660b91e67132d76680f88096f62" exitCode=0
Oct 30 19:23:28.836622 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:28.834352    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" event={"ID":"e3390d74-02de-49c0-92a6-3199b2f40395","Type":"ContainerDied","Data":"95076ca18b750ca2ca13659e828d1ba7d6e15660b91e67132d76680f88096f62"}
Oct 30 19:23:29.400103 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:29.400059    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" containerID="cri-o://602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911" gracePeriod=45
Oct 30 19:23:29.512135 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:29.512094    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main" containerID="cri-o://af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932" gracePeriod=45
Oct 30 19:23:29.686822 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:29.686737    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/health\": dial tcp 10.132.0.81:8000: connect: connection refused"
Oct 30 19:23:29.841854 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:29.841819    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" event={"ID":"e3390d74-02de-49c0-92a6-3199b2f40395","Type":"ContainerStarted","Data":"91302489b196d8d00bd36362ae8ec7ab85e988703e94114e7d6985b0a367f5f4"}
Oct 30 19:23:30.059135 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.059062    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podStartSLOduration=3.901829652 podStartE2EDuration="6.059042373s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.157212721s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:23:30.010637245 +0000 UTC m=+2792.091223467" watchObservedRunningTime="2026-10-30 19:23:30.059042373 +0000 UTC m=+2792.139628581"
Oct 30 19:23:30.169499 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.169474    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:23:30.317852 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.317762    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-home\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-home\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318020 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.317883    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tmp-dir\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318020 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.317889    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:30.318020 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.317934    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-model-cache\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318020 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.317970    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-dshm\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318020 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318011    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-cabundle-cert\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318048    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tls-certs\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318086    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kserve-provision-location\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318134    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kube-api-access-9jc66\" (UniqueName: \"kubernetes.io/projected/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kube-api-access-9jc66\") pod \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\" (UID: \"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116\") "
Oct 30 19:23:30.318294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318247    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-model-cache" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:30.318583 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318552    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.318583 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318566    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-home" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:30.318915 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.318884    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-cabundle-cert" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:23:30.321389 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.321351    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tls-certs" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:30.323127 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.323103    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-dshm" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:30.323433 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.323399    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kube-api-access-9jc66" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "kube-api-access-9jc66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:30.329219 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.329193    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tmp-dir" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:30.350086 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.350053    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kserve-provision-location" pod "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" (UID: "2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:23:30.419857 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.419824    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.419857 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.419852    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.419857 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.419860    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.420288 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.419869    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.420288 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.419878    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.420288 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.419887    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.420288 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.419896    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jc66\" (UniqueName: \"kubernetes.io/projected/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116-kube-api-access-9jc66\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:23:30.850074 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.850040    5025 generic.go:356] "Generic (PLEG): container finished" podID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerID="602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911" exitCode=0
Oct 30 19:23:30.850265 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.850091    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"
Oct 30 19:23:30.850265 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.850126    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" event={"ID":"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116","Type":"ContainerDied","Data":"602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911"}
Oct 30 19:23:30.850265 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.850180    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65" event={"ID":"2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116","Type":"ContainerDied","Data":"0b46edd01dff3db0bfb4d4fb0588e7b7856f0045c68a7ede639d829f9390313d"}
Oct 30 19:23:30.850265 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.850209    5025 scope.go:122] "RemoveContainer" containerID="602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911"
Oct 30 19:23:30.859679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.859660    5025 scope.go:122] "RemoveContainer" containerID="d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49"
Oct 30 19:23:30.868767 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.868745    5025 scope.go:122] "RemoveContainer" containerID="602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911"
Oct 30 19:23:30.869061 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:23:30.869040    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911\": container with ID starting with 602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911 not found: ID does not exist" containerID="602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911"
Oct 30 19:23:30.869147 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.869068    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911"} err="failed to get container status \"602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911\": rpc error: code = NotFound desc = could not find container \"602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911\": container with ID starting with 602f932f96c747a8dd0e0bbd1865f414142fe5fee94d383b3738e5b1d3345911 not found: ID does not exist"
Oct 30 19:23:30.869147 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.869087    5025 scope.go:122] "RemoveContainer" containerID="d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49"
Oct 30 19:23:30.869375 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:23:30.869359    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49\": container with ID starting with d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49 not found: ID does not exist" containerID="d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49"
Oct 30 19:23:30.869426 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:30.869378    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49"} err="failed to get container status \"d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49\": rpc error: code = NotFound desc = could not find container \"d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49\": container with ID starting with d944c97f0b95f0e93144d0d49ed0672a0b9e1e9d6ab31d7a4c46420eb72f0c49 not found: ID does not exist"
Oct 30 19:23:31.013201 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:31.013164    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"]
Oct 30 19:23:31.060593 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:31.060555    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-5pt65"]
Oct 30 19:23:32.056476 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:32.056435    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" path="/var/lib/kubelet/pods/2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116/volumes"
Oct 30 19:23:35.318275 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:35.318244    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:23:35.319776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:35.319733    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:23:38.181098 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:38.181062    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:45.319086 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:45.319032    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:23:48.181498 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:48.181458    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:55.318472 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:55.318423    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:23:58.082744 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:23:58.082707    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:23:58.180682 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:58.180645    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:23:58.180840 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:58.180749    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:23:59.767393 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.767356    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:23:59.911262 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.911224    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30e231f-b73b-4472-a7c0-75c2b75b33b9-kube-api-access-thzfr\" (UniqueName: \"kubernetes.io/projected/b30e231f-b73b-4472-a7c0-75c2b75b33b9-kube-api-access-thzfr\") pod \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") "
Oct 30 19:23:59.911493 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.911395    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30e231f-b73b-4472-a7c0-75c2b75b33b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/b30e231f-b73b-4472-a7c0-75c2b75b33b9-tls-certs\") pod \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\" (UID: \"b30e231f-b73b-4472-a7c0-75c2b75b33b9\") "
Oct 30 19:23:59.913492 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.913462    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30e231f-b73b-4472-a7c0-75c2b75b33b9-kube-api-access-thzfr" pod "b30e231f-b73b-4472-a7c0-75c2b75b33b9" (UID: "b30e231f-b73b-4472-a7c0-75c2b75b33b9"). InnerVolumeSpecName "kube-api-access-thzfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:23:59.913611 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.913592    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30e231f-b73b-4472-a7c0-75c2b75b33b9-tls-certs" pod "b30e231f-b73b-4472-a7c0-75c2b75b33b9" (UID: "b30e231f-b73b-4472-a7c0-75c2b75b33b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:23:59.995723 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.995683    5025 generic.go:356] "Generic (PLEG): container finished" podID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerID="af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932" exitCode=0
Oct 30 19:23:59.995887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.995737    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"
Oct 30 19:23:59.995887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.995772    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" event={"ID":"b30e231f-b73b-4472-a7c0-75c2b75b33b9","Type":"ContainerDied","Data":"af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932"}
Oct 30 19:23:59.995887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.995828    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668" event={"ID":"b30e231f-b73b-4472-a7c0-75c2b75b33b9","Type":"ContainerDied","Data":"de61fb6e8db1141590d521d3dd89f301be0d7a08db3c97a12ce2364abb5f0e51"}
Oct 30 19:23:59.995887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:23:59.995859    5025 scope.go:122] "RemoveContainer" containerID="af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932"
Oct 30 19:24:00.008081 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:00.007953    5025 scope.go:122] "RemoveContainer" containerID="af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932"
Oct 30 19:24:00.008446 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:24:00.008407    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932\": container with ID starting with af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932 not found: ID does not exist" containerID="af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932"
Oct 30 19:24:00.008564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:00.008458    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932"} err="failed to get container status \"af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932\": rpc error: code = NotFound desc = could not find container \"af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932\": container with ID starting with af61049a49a0a672fed9c959b1264450d80c02cee477bdfc0a8076a40dcbb932 not found: ID does not exist"
Oct 30 19:24:00.012982 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:00.012960    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thzfr\" (UniqueName: \"kubernetes.io/projected/b30e231f-b73b-4472-a7c0-75c2b75b33b9-kube-api-access-thzfr\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:00.013053 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:00.012985    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b30e231f-b73b-4472-a7c0-75c2b75b33b9-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:00.157103 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:00.157066    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"]
Oct 30 19:24:00.162105 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:00.162035    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6c5444664c-wp668"]
Oct 30 19:24:02.056286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:02.056252    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" path="/var/lib/kubelet/pods/b30e231f-b73b-4472-a7c0-75c2b75b33b9/volumes"
Oct 30 19:24:05.318815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:05.318779    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:24:15.318473 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:15.318372    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:24:25.318482 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:25.318432    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:24:26.943216 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943181    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"]
Oct 30 19:24:26.943626 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943599    5025 cpu_manager.go:414] "containerMap: removing container" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="storage-initializer"
Oct 30 19:24:26.943626 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943611    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="storage-initializer"
Oct 30 19:24:26.943626 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943621    5025 cpu_manager.go:414] "containerMap: removing container" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main"
Oct 30 19:24:26.943626 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943626    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main"
Oct 30 19:24:26.943776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943641    5025 cpu_manager.go:414] "containerMap: removing container" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main"
Oct 30 19:24:26.943776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943649    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main"
Oct 30 19:24:26.943776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943747    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30e231f-b73b-4472-a7c0-75c2b75b33b9" containerName="main"
Oct 30 19:24:26.943776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.943762    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2ffd15-ea29-48c6-a7fd-a4dddf7b5116" containerName="main"
Oct 30 19:24:26.946994 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.946975    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:26.994686 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:26.994648    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"]
Oct 30 19:24:27.038346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.038300    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Oct 30 19:24:27.040539 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.040517    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Oct 30 19:24:27.040539 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.040518    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:24:27.040699 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.040525    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Oct 30 19:24:27.040699 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.040566    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Oct 30 19:24:27.040699 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.040632    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-fzhfh\""
Oct 30 19:24:27.074604 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074570    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.074775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074611    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv62r\" (UniqueName: \"kubernetes.io/projected/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kube-api-access-dv62r\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.074775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074630    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.074775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074738    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.074775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074769    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.074945 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074789    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.074945 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074892    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.075027 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.074950    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5abd71f3-6201-4ec1-89d0-3d77c02beed1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176520 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176480    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176539    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5abd71f3-6201-4ec1-89d0-3d77c02beed1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176606    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176775 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176754    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv62r\" (UniqueName: \"kubernetes.io/projected/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kube-api-access-dv62r\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176833 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176796    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176890 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176847    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176947 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176896    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.176947 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176927    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.177055 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.176938    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.177055 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.177001    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.177299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.177272    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.177605 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.177580    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.177771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.177588    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5abd71f3-6201-4ec1-89d0-3d77c02beed1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.179231 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.179212    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.179553 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.179531    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.234458 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.234369    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv62r\" (UniqueName: \"kubernetes.io/projected/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kube-api-access-dv62r\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.263188 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.263153    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:27.366579 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.366545    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"]
Oct 30 19:24:27.370382 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.370360    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.424856 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.424820    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-9r7fn\""
Oct 30 19:24:27.426920 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.426884    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"]
Oct 30 19:24:27.479576 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.479539    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17ee1a5-208b-49ba-b476-58deaacf19b8-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.479779 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.479594    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2sc\" (UniqueName: \"kubernetes.io/projected/d17ee1a5-208b-49ba-b476-58deaacf19b8-kube-api-access-dx2sc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.581127 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.581032    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17ee1a5-208b-49ba-b476-58deaacf19b8-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.581127 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.581097    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2sc\" (UniqueName: \"kubernetes.io/projected/d17ee1a5-208b-49ba-b476-58deaacf19b8-kube-api-access-dx2sc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.584129 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.584101    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17ee1a5-208b-49ba-b476-58deaacf19b8-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.642038 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.642004    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2sc\" (UniqueName: \"kubernetes.io/projected/d17ee1a5-208b-49ba-b476-58deaacf19b8-kube-api-access-dx2sc\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.683941 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.683904    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:27.750064 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:27.750029    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"]
Oct 30 19:24:27.754907 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:24:27.754875    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abd71f3_6201_4ec1_89d0_3d77c02beed1.slice/crio-fe71d653507d13526f8978500ad8d2665b9783b52719924e8310f9fabb0d5956 WatchSource:0}: Error finding container fe71d653507d13526f8978500ad8d2665b9783b52719924e8310f9fabb0d5956: Status 404 returned error can't find the container with id fe71d653507d13526f8978500ad8d2665b9783b52719924e8310f9fabb0d5956
Oct 30 19:24:28.059463 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:28.059432    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"]
Oct 30 19:24:28.064356 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:24:28.064309    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17ee1a5_208b_49ba_b476_58deaacf19b8.slice/crio-05c59575a96caa93775b027c26365e1c0fa05a3b5cd67e13de8b70d25b63763b WatchSource:0}: Error finding container 05c59575a96caa93775b027c26365e1c0fa05a3b5cd67e13de8b70d25b63763b: Status 404 returned error can't find the container with id 05c59575a96caa93775b027c26365e1c0fa05a3b5cd67e13de8b70d25b63763b
Oct 30 19:24:28.090640 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:24:28.090609    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:24:28.145310 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:28.145264    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" event={"ID":"5abd71f3-6201-4ec1-89d0-3d77c02beed1","Type":"ContainerStarted","Data":"8f01ce381214a5fac36b46a081d8afdcc0fafe8c80573e086c06bbb31e572210"}
Oct 30 19:24:28.145496 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:28.145370    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" event={"ID":"5abd71f3-6201-4ec1-89d0-3d77c02beed1","Type":"ContainerStarted","Data":"fe71d653507d13526f8978500ad8d2665b9783b52719924e8310f9fabb0d5956"}
Oct 30 19:24:28.147178 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:28.147148    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" event={"ID":"d17ee1a5-208b-49ba-b476-58deaacf19b8","Type":"ContainerStarted","Data":"71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c"}
Oct 30 19:24:28.147294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:28.147206    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:24:28.147294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:28.147225    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" event={"ID":"d17ee1a5-208b-49ba-b476-58deaacf19b8","Type":"ContainerStarted","Data":"05c59575a96caa93775b027c26365e1c0fa05a3b5cd67e13de8b70d25b63763b"}
Oct 30 19:24:28.376574 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:28.376523    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" podStartSLOduration=1.376508043 podStartE2EDuration="1.376508043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:28.37510274 +0000 UTC m=+2850.455688966" watchObservedRunningTime="2026-10-30 19:24:28.376508043 +0000 UTC m=+2850.457094284"
Oct 30 19:24:31.167659 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:31.167622    5025 generic.go:356] "Generic (PLEG): container finished" podID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerID="8f01ce381214a5fac36b46a081d8afdcc0fafe8c80573e086c06bbb31e572210" exitCode=0
Oct 30 19:24:31.168337 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:31.168290    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" event={"ID":"5abd71f3-6201-4ec1-89d0-3d77c02beed1","Type":"ContainerDied","Data":"8f01ce381214a5fac36b46a081d8afdcc0fafe8c80573e086c06bbb31e572210"}
Oct 30 19:24:32.176217 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:32.176177    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" event={"ID":"5abd71f3-6201-4ec1-89d0-3d77c02beed1","Type":"ContainerStarted","Data":"e5a33309592fcf871d5264aa31eb5267a9e81e72e91f294515d3f610cfe23449"}
Oct 30 19:24:32.387365 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:32.387281    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podStartSLOduration=3.980150609 podStartE2EDuration="6.387260037s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.407109428s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:24:32.339224002 +0000 UTC m=+2854.419810209" watchObservedRunningTime="2026-10-30 19:24:32.387260037 +0000 UTC m=+2854.467846246"
Oct 30 19:24:33.133258 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:33.133223    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"]
Oct 30 19:24:33.133462 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:33.133437    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy" containerID="cri-o://8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9" gracePeriod=30
Oct 30 19:24:35.319042 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:35.318989    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:24:37.263572 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:37.263529    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:37.264071 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:37.263617    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:24:37.265231 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:37.265198    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:24:37.982189 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:37.982151    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:24:37.982388 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:37.982223    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:24:38.404926 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.404890    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:24:38.494015 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.493970    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-certs\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494215 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494048    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-podinfo\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494215 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494120    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-kube-api-access-8pngz\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-kube-api-access-8pngz\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494215 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494192    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-socket\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494237    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4091c6a4-fecc-45a4-92ad-ce82248ea675-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4091c6a4-fecc-45a4-92ad-ce82248ea675-istiod-ca-cert\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494252    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-certs" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.494413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494268    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-credential-socket\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494362    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-data\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494627 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494427    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-envoy\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494627 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494454    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-token\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-token\") pod \"4091c6a4-fecc-45a4-92ad-ce82248ea675\" (UID: \"4091c6a4-fecc-45a4-92ad-ce82248ea675\") "
Oct 30 19:24:38.494627 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494506    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-socket" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.494627 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494620    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4091c6a4-fecc-45a4-92ad-ce82248ea675-istiod-ca-cert" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:24:38.494850 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494753    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-credential-socket" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.494952 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494926    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-data" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.495026 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494978    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.495026 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.494999    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.495026 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.495015    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4091c6a4-fecc-45a4-92ad-ce82248ea675-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.495252 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.495034    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.496940 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.496907    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-envoy" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:24:38.497049 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.496915    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-kube-api-access-8pngz" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "kube-api-access-8pngz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.497049 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.496915    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-podinfo" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:24:38.497218 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.497202    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-token" pod "4091c6a4-fecc-45a4-92ad-ce82248ea675" (UID: "4091c6a4-fecc-45a4-92ad-ce82248ea675"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:24:38.596036 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.595994    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pngz\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-kube-api-access-8pngz\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.596036 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.596035    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.596264 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.596050    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.596264 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.596115    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:38.596264 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:38.596132    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4091c6a4-fecc-45a4-92ad-ce82248ea675-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:24:39.215818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.215780    5025 generic.go:356] "Generic (PLEG): container finished" podID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerID="8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9" exitCode=0
Oct 30 19:24:39.215818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.215826    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"
Oct 30 19:24:39.216033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.215874    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" event={"ID":"4091c6a4-fecc-45a4-92ad-ce82248ea675","Type":"ContainerDied","Data":"8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9"}
Oct 30 19:24:39.216033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.215938    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw" event={"ID":"4091c6a4-fecc-45a4-92ad-ce82248ea675","Type":"ContainerDied","Data":"0877565ddf6eeb749e16518b97ecb419acf49b7786e1db0e3d7259a738997ea7"}
Oct 30 19:24:39.216033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.215974    5025 scope.go:122] "RemoveContainer" containerID="8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9"
Oct 30 19:24:39.227776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.227759    5025 scope.go:122] "RemoveContainer" containerID="8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9"
Oct 30 19:24:39.228078 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:24:39.228055    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9\": container with ID starting with 8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9 not found: ID does not exist" containerID="8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9"
Oct 30 19:24:39.228173 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.228084    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9"} err="failed to get container status \"8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9\": rpc error: code = NotFound desc = could not find container \"8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9\": container with ID starting with 8cb1f1621aaa4850b33f79aa25d505e4d6d7c3c02dd2d31627af48aa2e3c05c9 not found: ID does not exist"
Oct 30 19:24:39.427818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.427769    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"]
Oct 30 19:24:39.473352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:39.473250    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-nhvlw"]
Oct 30 19:24:40.057968 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:40.057928    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" path="/var/lib/kubelet/pods/4091c6a4-fecc-45a4-92ad-ce82248ea675/volumes"
Oct 30 19:24:45.318771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:45.318732    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:24:47.263655 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:47.263605    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:24:55.318542 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:55.318491    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:24:57.263837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:57.263784    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:24:58.082289 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:24:58.082248    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:24:59.154781 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:24:59.154745    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:25:05.318376 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:05.318312    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:25:07.264027 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:07.263976    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:25:15.319118 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:15.319053    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:25:17.264644 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:17.264602    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:25:25.328732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:25.328699    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:25:25.340464 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:25.340438    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:25:27.264592 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:27.264543    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:25:28.083047 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:25:28.083008    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:25:32.643969 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:32.643929    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"]
Oct 30 19:25:33.629536 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:33.629504    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"]
Oct 30 19:25:33.629726 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:33.629700    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy" containerID="cri-o://e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43" gracePeriod=30
Oct 30 19:25:37.263839 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:37.263743    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:25:38.103011 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.102972    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-92qxs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:25:38.103304 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.103034    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:25:38.887300 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.887275    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:25:38.977302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977215    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-socket\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977279    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-data\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977308    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-credential-socket\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977359    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4d6a566-5379-474e-8776-3cad924206a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d6a566-5379-474e-8776-3cad924206a2-istio-podinfo\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977409    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-certs\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977438    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-istio-token\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-istio-token\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977482    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d6a566-5379-474e-8776-3cad924206a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d6a566-5379-474e-8776-3cad924206a2-istiod-ca-cert\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977520    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-envoy\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977579    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-socket" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:38.977837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977621    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-kube-api-access-h49ht\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-kube-api-access-h49ht\") pod \"c4d6a566-5379-474e-8776-3cad924206a2\" (UID: \"c4d6a566-5379-474e-8776-3cad924206a2\") "
Oct 30 19:25:38.977837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977647    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-data" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:38.977837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977658    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-credential-socket" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:38.978037 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977909    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d6a566-5379-474e-8776-3cad924206a2-istiod-ca-cert" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:38.978037 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.977985    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:38.978037 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.978004    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:38.978037 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.978020    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:38.978265 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.978231    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-certs" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:38.979951 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.979914    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4d6a566-5379-474e-8776-3cad924206a2-istio-podinfo" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:25:38.980255 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.980225    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-envoy" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:38.980255 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.980244    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-kube-api-access-h49ht" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "kube-api-access-h49ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:38.980415 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:38.980224    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-istio-token" pod "c4d6a566-5379-474e-8776-3cad924206a2" (UID: "c4d6a566-5379-474e-8776-3cad924206a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:39.079201 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.079161    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h49ht\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-kube-api-access-h49ht\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.079201 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.079200    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d6a566-5379-474e-8776-3cad924206a2-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.079438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.079215    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.079438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.079229    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d6a566-5379-474e-8776-3cad924206a2-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.079438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.079244    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d6a566-5379-474e-8776-3cad924206a2-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.079438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.079257    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d6a566-5379-474e-8776-3cad924206a2-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:39.527364 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.527313    5025 generic.go:356] "Generic (PLEG): container finished" podID="c4d6a566-5379-474e-8776-3cad924206a2" containerID="e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43" exitCode=0
Oct 30 19:25:39.527545 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.527383    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" event={"ID":"c4d6a566-5379-474e-8776-3cad924206a2","Type":"ContainerDied","Data":"e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43"}
Oct 30 19:25:39.527545 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.527389    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"
Oct 30 19:25:39.527545 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.527450    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs" event={"ID":"c4d6a566-5379-474e-8776-3cad924206a2","Type":"ContainerDied","Data":"41df6993c10e415f6a82af20d839c17fed4628e9fbde540bd548d0ff482ac4b2"}
Oct 30 19:25:39.527545 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.527472    5025 scope.go:122] "RemoveContainer" containerID="e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43"
Oct 30 19:25:39.537681 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.537659    5025 scope.go:122] "RemoveContainer" containerID="e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43"
Oct 30 19:25:39.537996 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:25:39.537973    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43\": container with ID starting with e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43 not found: ID does not exist" containerID="e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43"
Oct 30 19:25:39.538082 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.538006    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43"} err="failed to get container status \"e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43\": rpc error: code = NotFound desc = could not find container \"e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43\": container with ID starting with e685c2473c7958e0536c67858ce4e4bde41b38b6f49cd575a716599fef3c2a43 not found: ID does not exist"
Oct 30 19:25:39.687110 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.687073    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"]
Oct 30 19:25:39.734266 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:39.734232    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-92qxs"]
Oct 30 19:25:40.057382 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:40.057341    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d6a566-5379-474e-8776-3cad924206a2" path="/var/lib/kubelet/pods/c4d6a566-5379-474e-8776-3cad924206a2/volumes"
Oct 30 19:25:47.264221 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:47.264171    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:25:47.672956 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:47.672914    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" containerID="cri-o://91302489b196d8d00bd36362ae8ec7ab85e988703e94114e7d6985b0a367f5f4" gracePeriod=45
Oct 30 19:25:48.329747 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.329692    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": dial tcp 10.132.0.83:8000: connect: connection refused"
Oct 30 19:25:48.579338 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.579296    5025 generic.go:356] "Generic (PLEG): container finished" podID="e3390d74-02de-49c0-92a6-3199b2f40395" containerID="91302489b196d8d00bd36362ae8ec7ab85e988703e94114e7d6985b0a367f5f4" exitCode=0
Oct 30 19:25:48.579500 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.579361    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" event={"ID":"e3390d74-02de-49c0-92a6-3199b2f40395","Type":"ContainerDied","Data":"91302489b196d8d00bd36362ae8ec7ab85e988703e94114e7d6985b0a367f5f4"}
Oct 30 19:25:48.630912 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.630880    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:25:48.662874 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.662843    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-model-cache\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663014 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.662893    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-kserve-provision-location\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663014 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.662927    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3390d74-02de-49c0-92a6-3199b2f40395-kube-api-access-f9n5h\" (UniqueName: \"kubernetes.io/projected/e3390d74-02de-49c0-92a6-3199b2f40395-kube-api-access-f9n5h\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663014 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.662966    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-home\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-home\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663014 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663002    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-dshm\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663025    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3390d74-02de-49c0-92a6-3199b2f40395-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3390d74-02de-49c0-92a6-3199b2f40395-tls-certs\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663057    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-tmp-dir\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663088    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3390d74-02de-49c0-92a6-3199b2f40395-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3390d74-02de-49c0-92a6-3199b2f40395-cabundle-cert\") pod \"e3390d74-02de-49c0-92a6-3199b2f40395\" (UID: \"e3390d74-02de-49c0-92a6-3199b2f40395\") "
Oct 30 19:25:48.663230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663136    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-model-cache" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:48.663447 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663382    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:48.663527 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663502    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3390d74-02de-49c0-92a6-3199b2f40395-cabundle-cert" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:25:48.663848 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.663813    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-home" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:48.665934 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.665904    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3390d74-02de-49c0-92a6-3199b2f40395-kube-api-access-f9n5h" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "kube-api-access-f9n5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:25:48.665934 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.665919    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3390d74-02de-49c0-92a6-3199b2f40395-tls-certs" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:25:48.666060 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.665933    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-dshm" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:48.682268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.682232    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-tmp-dir" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:48.682896 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.682876    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-kserve-provision-location" pod "e3390d74-02de-49c0-92a6-3199b2f40395" (UID: "e3390d74-02de-49c0-92a6-3199b2f40395"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:25:48.764769 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.764734    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:48.764769 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.764765    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3390d74-02de-49c0-92a6-3199b2f40395-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:48.764928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.764783    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:48.764928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.764797    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e3390d74-02de-49c0-92a6-3199b2f40395-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:48.764928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.764812    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:48.764928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.764828    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9n5h\" (UniqueName: \"kubernetes.io/projected/e3390d74-02de-49c0-92a6-3199b2f40395-kube-api-access-f9n5h\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:48.764928 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:48.764844    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3390d74-02de-49c0-92a6-3199b2f40395-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:25:49.586896 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:49.586867    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"
Oct 30 19:25:49.586896 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:49.586881    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp" event={"ID":"e3390d74-02de-49c0-92a6-3199b2f40395","Type":"ContainerDied","Data":"1a3b935bad88f93cfdd01a7dc5f0fab448ba093a199e8228c3c6e4f8d6585f3f"}
Oct 30 19:25:49.587467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:49.586962    5025 scope.go:122] "RemoveContainer" containerID="91302489b196d8d00bd36362ae8ec7ab85e988703e94114e7d6985b0a367f5f4"
Oct 30 19:25:49.596208 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:49.596192    5025 scope.go:122] "RemoveContainer" containerID="95076ca18b750ca2ca13659e828d1ba7d6e15660b91e67132d76680f88096f62"
Oct 30 19:25:49.795109 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:49.795074    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"]
Oct 30 19:25:49.842572 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:49.842505    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-98446cffd-6sfgp"]
Oct 30 19:25:50.056726 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:50.056685    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" path="/var/lib/kubelet/pods/e3390d74-02de-49c0-92a6-3199b2f40395/volumes"
Oct 30 19:25:57.263998 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:25:57.263954    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:25:58.082816 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:25:58.082778    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:26:07.264212 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:07.264164    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:26:17.264637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:17.264590    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Oct 30 19:26:27.274839 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:27.274806    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:26:27.282575 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:27.282541    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:26:28.084933 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:26:28.084896    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:26:42.200408 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:42.200375    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"]
Oct 30 19:26:42.216328 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:42.216287    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"]
Oct 30 19:26:49.154922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:49.154884    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:26:57.223156 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:57.223118    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main" containerID="cri-o://e5a33309592fcf871d5264aa31eb5267a9e81e72e91f294515d3f610cfe23449" gracePeriod=45
Oct 30 19:26:57.235428 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:57.235394    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main" containerID="cri-o://71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c" gracePeriod=45
Oct 30 19:26:57.919126 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:57.919092    5025 generic.go:356] "Generic (PLEG): container finished" podID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerID="e5a33309592fcf871d5264aa31eb5267a9e81e72e91f294515d3f610cfe23449" exitCode=0
Oct 30 19:26:57.919265 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:57.919159    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" event={"ID":"5abd71f3-6201-4ec1-89d0-3d77c02beed1","Type":"ContainerDied","Data":"e5a33309592fcf871d5264aa31eb5267a9e81e72e91f294515d3f610cfe23449"}
Oct 30 19:26:57.977070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:57.977048    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:26:58.081968 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:26:58.081894    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:26:58.094426 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094404    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-home\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-home\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.094544 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094466    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tls-certs\" (UniqueName: \"kubernetes.io/secret/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tls-certs\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.094544 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094503    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kube-api-access-dv62r\" (UniqueName: \"kubernetes.io/projected/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kube-api-access-dv62r\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.094660 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094586    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kserve-provision-location\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.094660 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094620    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tmp-dir\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.094660 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094653    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5abd71f3-6201-4ec1-89d0-3d77c02beed1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5abd71f3-6201-4ec1-89d0-3d77c02beed1-cabundle-cert\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.094820 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094691    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-dshm\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.094820 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.094737    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-model-cache\") pod \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\" (UID: \"5abd71f3-6201-4ec1-89d0-3d77c02beed1\") "
Oct 30 19:26:58.095184 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.095156    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5abd71f3-6201-4ec1-89d0-3d77c02beed1-cabundle-cert" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:26:58.095502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.095266    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-home" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.095502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.095296    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-model-cache" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.096835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.096799    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tls-certs" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:26:58.096835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.096809    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kube-api-access-dv62r" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "kube-api-access-dv62r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:26:58.097874 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.097849    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-dshm" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.110335 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.110295    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tmp-dir" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.118835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.118810    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kserve-provision-location" pod "5abd71f3-6201-4ec1-89d0-3d77c02beed1" (UID: "5abd71f3-6201-4ec1-89d0-3d77c02beed1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:26:58.196474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196402    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.196474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196440    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.196474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196456    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5abd71f3-6201-4ec1-89d0-3d77c02beed1-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.196474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196470    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.196732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196484    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.196732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196498    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5abd71f3-6201-4ec1-89d0-3d77c02beed1-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.196732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196513    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5abd71f3-6201-4ec1-89d0-3d77c02beed1-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.196732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.196526    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv62r\" (UniqueName: \"kubernetes.io/projected/5abd71f3-6201-4ec1-89d0-3d77c02beed1-kube-api-access-dv62r\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:26:58.459171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.459140    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:26:58.463398 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.463366    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:26:58.925387 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.925356    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"
Oct 30 19:26:58.925591 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.925354    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf" event={"ID":"5abd71f3-6201-4ec1-89d0-3d77c02beed1","Type":"ContainerDied","Data":"fe71d653507d13526f8978500ad8d2665b9783b52719924e8310f9fabb0d5956"}
Oct 30 19:26:58.925591 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.925514    5025 scope.go:122] "RemoveContainer" containerID="e5a33309592fcf871d5264aa31eb5267a9e81e72e91f294515d3f610cfe23449"
Oct 30 19:26:58.934200 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:58.934034    5025 scope.go:122] "RemoveContainer" containerID="8f01ce381214a5fac36b46a081d8afdcc0fafe8c80573e086c06bbb31e572210"
Oct 30 19:26:59.086063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:59.086033    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"]
Oct 30 19:26:59.133528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:59.133500    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-54jjhqf"]
Oct 30 19:26:59.154706 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:26:59.154671    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:27:00.056157 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:00.056121    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" path="/var/lib/kubelet/pods/5abd71f3-6201-4ec1-89d0-3d77c02beed1/volumes"
Oct 30 19:27:09.154702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:09.154610    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:27:09.155233 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:09.154723    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:27:19.154529 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:19.154486    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:27:27.484295 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:27.484267    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:27:27.554034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:27.553938    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17ee1a5-208b-49ba-b476-58deaacf19b8-kube-api-access-dx2sc\" (UniqueName: \"kubernetes.io/projected/d17ee1a5-208b-49ba-b476-58deaacf19b8-kube-api-access-dx2sc\") pod \"d17ee1a5-208b-49ba-b476-58deaacf19b8\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") "
Oct 30 19:27:27.554034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:27.554011    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d17ee1a5-208b-49ba-b476-58deaacf19b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/d17ee1a5-208b-49ba-b476-58deaacf19b8-tls-certs\") pod \"d17ee1a5-208b-49ba-b476-58deaacf19b8\" (UID: \"d17ee1a5-208b-49ba-b476-58deaacf19b8\") "
Oct 30 19:27:27.556131 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:27.556100    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17ee1a5-208b-49ba-b476-58deaacf19b8-kube-api-access-dx2sc" pod "d17ee1a5-208b-49ba-b476-58deaacf19b8" (UID: "d17ee1a5-208b-49ba-b476-58deaacf19b8"). InnerVolumeSpecName "kube-api-access-dx2sc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:27:27.556131 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:27.556128    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d17ee1a5-208b-49ba-b476-58deaacf19b8-tls-certs" pod "d17ee1a5-208b-49ba-b476-58deaacf19b8" (UID: "d17ee1a5-208b-49ba-b476-58deaacf19b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:27:27.654887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:27.654850    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d17ee1a5-208b-49ba-b476-58deaacf19b8-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:27.654887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:27.654882    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx2sc\" (UniqueName: \"kubernetes.io/projected/d17ee1a5-208b-49ba-b476-58deaacf19b8-kube-api-access-dx2sc\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:27:28.056479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.056442    5025 generic.go:356] "Generic (PLEG): container finished" podID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerID="71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c" exitCode=0
Oct 30 19:27:28.056678 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.056540    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"
Oct 30 19:27:28.057011 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.056866    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" event={"ID":"d17ee1a5-208b-49ba-b476-58deaacf19b8","Type":"ContainerDied","Data":"71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c"}
Oct 30 19:27:28.057011 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.056923    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6" event={"ID":"d17ee1a5-208b-49ba-b476-58deaacf19b8","Type":"ContainerDied","Data":"05c59575a96caa93775b027c26365e1c0fa05a3b5cd67e13de8b70d25b63763b"}
Oct 30 19:27:28.057011 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.056954    5025 scope.go:122] "RemoveContainer" containerID="71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c"
Oct 30 19:27:28.067186 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.067165    5025 scope.go:122] "RemoveContainer" containerID="71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c"
Oct 30 19:27:28.067671 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:27:28.067639    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c\": container with ID starting with 71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c not found: ID does not exist" containerID="71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c"
Oct 30 19:27:28.067743 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.067692    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c"} err="failed to get container status \"71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c\": rpc error: code = NotFound desc = could not find container \"71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c\": container with ID starting with 71844cc7d78ecdc01c4f24a5b661030f6834efa8d94f7be87ebd628d7cd3c72c not found: ID does not exist"
Oct 30 19:27:28.082807 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:27:28.082778    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:27:28.215367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.215332    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"]
Oct 30 19:27:28.262726 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:28.262695    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schezq9k6"]
Oct 30 19:27:30.056290 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:30.056259    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" path="/var/lib/kubelet/pods/d17ee1a5-208b-49ba-b476-58deaacf19b8/volumes"
Oct 30 19:27:45.210559 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210525    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"]
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210893    5025 cpu_manager.go:414] "containerMap: removing container" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210908    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210919    5025 cpu_manager.go:414] "containerMap: removing container" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="storage-initializer"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210925    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="storage-initializer"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210940    5025 cpu_manager.go:414] "containerMap: removing container" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210946    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210958    5025 cpu_manager.go:414] "containerMap: removing container" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="storage-initializer"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210962    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="storage-initializer"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210977    5025 cpu_manager.go:414] "containerMap: removing container" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210985    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.210998    5025 cpu_manager.go:414] "containerMap: removing container" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211003    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211013    5025 cpu_manager.go:414] "containerMap: removing container" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main"
Oct 30 19:27:45.211031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211017    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main"
Oct 30 19:27:45.211474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211082    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17ee1a5-208b-49ba-b476-58deaacf19b8" containerName="main"
Oct 30 19:27:45.211474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211090    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="5abd71f3-6201-4ec1-89d0-3d77c02beed1" containerName="main"
Oct 30 19:27:45.211474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211098    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3390d74-02de-49c0-92a6-3199b2f40395" containerName="main"
Oct 30 19:27:45.211474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211104    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="4091c6a4-fecc-45a4-92ad-ce82248ea675" containerName="istio-proxy"
Oct 30 19:27:45.211474 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.211111    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d6a566-5379-474e-8776-3cad924206a2" containerName="istio-proxy"
Oct 30 19:27:45.215997 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.215976    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.267295 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.267268    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:27:45.267438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.267405    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Oct 30 19:27:45.307388 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307355    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-model-cache\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.307548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307395    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/575d39b3-46b1-4660-9203-93afaf4de0f2-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.307548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307426    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-home\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.307548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307535    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-dshm\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.307688 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307590    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.307688 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307645    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.307770 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307702    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575d39b3-46b1-4660-9203-93afaf4de0f2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.307770 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.307730    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mll5x\" (UniqueName: \"kubernetes.io/projected/575d39b3-46b1-4660-9203-93afaf4de0f2-kube-api-access-mll5x\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.313977 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.313955    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Oct 30 19:27:45.313977 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.313980    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-76lk6\""
Oct 30 19:27:45.314237 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.314215    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Oct 30 19:27:45.314286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.314232    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Oct 30 19:27:45.314286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.314251    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-fhftj\""
Oct 30 19:27:45.317118 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.317097    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"]
Oct 30 19:27:45.409149 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409115    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-model-cache\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409149 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409149    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/575d39b3-46b1-4660-9203-93afaf4de0f2-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409177    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-home\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409214    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-dshm\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409257    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409276    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409298    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575d39b3-46b1-4660-9203-93afaf4de0f2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409337    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mll5x\" (UniqueName: \"kubernetes.io/projected/575d39b3-46b1-4660-9203-93afaf4de0f2-kube-api-access-mll5x\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409805 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409636    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-model-cache\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409805 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409673    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-home\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409916    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.409995 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409944    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.410053 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.409996    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/575d39b3-46b1-4660-9203-93afaf4de0f2-cabundle-cert\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.411947 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.411914    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-dshm\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.412294 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.412272    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575d39b3-46b1-4660-9203-93afaf4de0f2-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.464929 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.464862    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mll5x\" (UniqueName: \"kubernetes.io/projected/575d39b3-46b1-4660-9203-93afaf4de0f2-kube-api-access-mll5x\") pod \"custom-route-timeout-pd-test-kserve-694879b7f-n8gx5\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.531887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.531858    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:45.960035 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:45.960005    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"]
Oct 30 19:27:45.965803 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:27:45.965774    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod575d39b3_46b1_4660_9203_93afaf4de0f2.slice/crio-b187ce3b8aee70462cd98b12796201453ff31b524136edfbc5e5527a32253b1c WatchSource:0}: Error finding container b187ce3b8aee70462cd98b12796201453ff31b524136edfbc5e5527a32253b1c: Status 404 returned error can't find the container with id b187ce3b8aee70462cd98b12796201453ff31b524136edfbc5e5527a32253b1c
Oct 30 19:27:46.136989 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:46.136952    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerStarted","Data":"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"}
Oct 30 19:27:46.137182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:46.137019    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerStarted","Data":"b187ce3b8aee70462cd98b12796201453ff31b524136edfbc5e5527a32253b1c"}
Oct 30 19:27:46.137182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:46.137059    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:47.143130 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:47.143093    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerStarted","Data":"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"}
Oct 30 19:27:49.154870 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:49.154837    5025 generic.go:356] "Generic (PLEG): container finished" podID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerID="be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9" exitCode=0
Oct 30 19:27:49.155303 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:49.154914    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerDied","Data":"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"}
Oct 30 19:27:50.162980 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:50.162941    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerStarted","Data":"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"}
Oct 30 19:27:50.380194 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:50.380138    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podStartSLOduration=3.179139809 podStartE2EDuration="5.380123337s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.200983528s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:27:50.378248553 +0000 UTC m=+3052.458834759" watchObservedRunningTime="2026-10-30 19:27:50.380123337 +0000 UTC m=+3052.460709544"
Oct 30 19:27:55.532651 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:55.532606    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:55.533197 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:55.532681    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:27:55.533852 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:27:55.533824    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:27:58.083252 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:27:58.083205    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:28:05.532955 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:28:05.532914    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:28:05.545934 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:28:05.545904    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:28:15.533096 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:28:15.533046    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:28:25.532602 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:28:25.532550    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:28:28.086886 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:28:28.086849    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:28:35.532705 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:28:35.532654    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:28:45.532605 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:28:45.532491    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:28:55.532493 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:28:55.532446    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:28:58.082164 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:28:58.082118    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:29:05.532452 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:29:05.532386    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:29:15.533170 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:29:15.533124    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:29:25.532339 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:29:25.532270    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:29:28.082501 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:29:28.082453    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:29:35.532472 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:29:35.532425    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.86:8001/health\": dial tcp 10.132.0.86:8001: connect: connection refused"
Oct 30 19:29:45.541674 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:29:45.541642    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:29:45.569647 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:29:45.569621    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:29:58.083196 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:29:58.083156    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:29:58.590615 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:29:58.590582    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"]
Oct 30 19:30:13.617611 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:13.617494    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main" containerID="cri-o://b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9" gracePeriod=45
Oct 30 19:30:14.234255 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.234198    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="llm-d-routing-sidecar" containerID="cri-o://db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba" gracePeriod=45
Oct 30 19:30:14.490646 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.490619    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:30:14.652815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652741    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-kserve-provision-location\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.652815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652787    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-model-cache\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652825    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/575d39b3-46b1-4660-9203-93afaf4de0f2-tls-certs\" (UniqueName: \"kubernetes.io/secret/575d39b3-46b1-4660-9203-93afaf4de0f2-tls-certs\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652854    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-home\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-home\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652882    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-dshm\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652918    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/575d39b3-46b1-4660-9203-93afaf4de0f2-kube-api-access-mll5x\" (UniqueName: \"kubernetes.io/projected/575d39b3-46b1-4660-9203-93afaf4de0f2-kube-api-access-mll5x\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652954    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-tmp-dir\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.652993    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/575d39b3-46b1-4660-9203-93afaf4de0f2-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/575d39b3-46b1-4660-9203-93afaf4de0f2-cabundle-cert\") pod \"575d39b3-46b1-4660-9203-93afaf4de0f2\" (UID: \"575d39b3-46b1-4660-9203-93afaf4de0f2\") "
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.653155    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-model-cache" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.653352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.653308    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.654357 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.653934    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/575d39b3-46b1-4660-9203-93afaf4de0f2-cabundle-cert" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:30:14.654357 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.653949    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-home" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.656273 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.656237    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/575d39b3-46b1-4660-9203-93afaf4de0f2-tls-certs" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:30:14.656400 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.656334    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575d39b3-46b1-4660-9203-93afaf4de0f2-kube-api-access-mll5x" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "kube-api-access-mll5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:30:14.656735 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.656716    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-dshm" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.671314 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.671288    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-tmp-dir" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.679929 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.679904    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-kserve-provision-location" pod "575d39b3-46b1-4660-9203-93afaf4de0f2" (UID: "575d39b3-46b1-4660-9203-93afaf4de0f2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:30:14.754143 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.754118    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.754143 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.754140    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/575d39b3-46b1-4660-9203-93afaf4de0f2-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.754292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.754151    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.754292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.754159    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.754292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.754168    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mll5x\" (UniqueName: \"kubernetes.io/projected/575d39b3-46b1-4660-9203-93afaf4de0f2-kube-api-access-mll5x\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.754292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.754176    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/575d39b3-46b1-4660-9203-93afaf4de0f2-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.754292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.754184    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/575d39b3-46b1-4660-9203-93afaf4de0f2-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:30:14.856399 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.856364    5025 generic.go:356] "Generic (PLEG): container finished" podID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerID="b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9" exitCode=0
Oct 30 19:30:14.856399 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.856394    5025 generic.go:356] "Generic (PLEG): container finished" podID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerID="db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba" exitCode=0
Oct 30 19:30:14.856577 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.856432    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"
Oct 30 19:30:14.856577 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.856432    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerDied","Data":"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"}
Oct 30 19:30:14.856577 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.856554    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerDied","Data":"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"}
Oct 30 19:30:14.856707 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.856582    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5" event={"ID":"575d39b3-46b1-4660-9203-93afaf4de0f2","Type":"ContainerDied","Data":"b187ce3b8aee70462cd98b12796201453ff31b524136edfbc5e5527a32253b1c"}
Oct 30 19:30:14.856707 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.856611    5025 scope.go:122] "RemoveContainer" containerID="b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"
Oct 30 19:30:14.865599 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.865580    5025 scope.go:122] "RemoveContainer" containerID="be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"
Oct 30 19:30:14.873607 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.873588    5025 scope.go:122] "RemoveContainer" containerID="db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"
Oct 30 19:30:14.881119 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.881101    5025 scope.go:122] "RemoveContainer" containerID="b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"
Oct 30 19:30:14.881391 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:30:14.881367    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9\": container with ID starting with b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9 not found: ID does not exist" containerID="b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"
Oct 30 19:30:14.881506 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.881399    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"} err="failed to get container status \"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9\": rpc error: code = NotFound desc = could not find container \"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9\": container with ID starting with b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9 not found: ID does not exist"
Oct 30 19:30:14.881506 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.881420    5025 scope.go:122] "RemoveContainer" containerID="be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"
Oct 30 19:30:14.881708 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:30:14.881681    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9\": container with ID starting with be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9 not found: ID does not exist" containerID="be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"
Oct 30 19:30:14.881748 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.881715    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"} err="failed to get container status \"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9\": rpc error: code = NotFound desc = could not find container \"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9\": container with ID starting with be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9 not found: ID does not exist"
Oct 30 19:30:14.881748 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.881730    5025 scope.go:122] "RemoveContainer" containerID="db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"
Oct 30 19:30:14.881987 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:30:14.881965    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba\": container with ID starting with db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba not found: ID does not exist" containerID="db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"
Oct 30 19:30:14.882051 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.881994    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"} err="failed to get container status \"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba\": rpc error: code = NotFound desc = could not find container \"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba\": container with ID starting with db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba not found: ID does not exist"
Oct 30 19:30:14.882051 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.882008    5025 scope.go:122] "RemoveContainer" containerID="b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"
Oct 30 19:30:14.882256 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.882235    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9"} err="failed to get container status \"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9\": rpc error: code = NotFound desc = could not find container \"b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9\": container with ID starting with b964837d98eba0c23c6c087a3dae57bd03dd879b40efc6d9ed53d20664ad87c9 not found: ID does not exist"
Oct 30 19:30:14.882352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.882256    5025 scope.go:122] "RemoveContainer" containerID="be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"
Oct 30 19:30:14.882536 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.882513    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9"} err="failed to get container status \"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9\": rpc error: code = NotFound desc = could not find container \"be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9\": container with ID starting with be4e51eca07da0519ba6839e710b3db75da88a110bec3a70d179c7fb97dd07d9 not found: ID does not exist"
Oct 30 19:30:14.882612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.882538    5025 scope.go:122] "RemoveContainer" containerID="db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"
Oct 30 19:30:14.882779 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:14.882759    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba"} err="failed to get container status \"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba\": rpc error: code = NotFound desc = could not find container \"db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba\": container with ID starting with db682baff7471abbb0df112917cc88ef10a7716ab62fd2604eaa49eddc9cafba not found: ID does not exist"
Oct 30 19:30:15.017826 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:15.017750    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"]
Oct 30 19:30:15.023247 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:15.023227    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-694879b7f-n8gx5"]
Oct 30 19:30:16.061557 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:16.061520    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" path="/var/lib/kubelet/pods/575d39b3-46b1-4660-9203-93afaf4de0f2/volumes"
Oct 30 19:30:28.081681 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:30:28.081647    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:30:48.671986 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.671952    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"]
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672484    5025 cpu_manager.go:414] "containerMap: removing container" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main"
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672504    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main"
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672534    5025 cpu_manager.go:414] "containerMap: removing container" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="llm-d-routing-sidecar"
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672542    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="llm-d-routing-sidecar"
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672555    5025 cpu_manager.go:414] "containerMap: removing container" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="storage-initializer"
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672561    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="storage-initializer"
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672706    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="main"
Oct 30 19:30:48.672938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.672722    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="575d39b3-46b1-4660-9203-93afaf4de0f2" containerName="llm-d-routing-sidecar"
Oct 30 19:30:48.676078 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.676059    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.727235 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.727200    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Oct 30 19:30:48.749582 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749548    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-credential-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749706 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749590    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749706 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749606    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99c339-6d14-4c35-856b-efebf44cc926-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749784 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749733    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-certs\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749827 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749781    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-envoy\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749872 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749846    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99c339-6d14-4c35-856b-efebf44cc926-istio-podinfo\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749907 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749877    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mldtp\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-kube-api-access-mldtp\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749907 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749893    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-data\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.749984 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.749910    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-istio-token\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.750634 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.750617    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Oct 30 19:30:48.765350 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.765306    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:30:48.765459 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.765393    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-gateway-2-openshift-default-dockercfg-5szg8\""
Oct 30 19:30:48.775367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.775345    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"]
Oct 30 19:30:48.850520 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850488    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-envoy\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.850692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850538    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99c339-6d14-4c35-856b-efebf44cc926-istio-podinfo\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.850692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850576    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mldtp\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-kube-api-access-mldtp\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.850692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850593    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-data\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.850692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850612    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-istio-token\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.850692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850631    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-credential-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.850692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850656    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.850692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850672    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99c339-6d14-4c35-856b-efebf44cc926-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.851033 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.850716    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-certs\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.851098 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.851055    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-data\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.851098 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.851087    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-credential-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.851187 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.851124    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-certs\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.851187 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.851166    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-socket\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.851644 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.851623    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99c339-6d14-4c35-856b-efebf44cc926-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.853438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.853418    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-envoy\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.853658 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.853637    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99c339-6d14-4c35-856b-efebf44cc926-istio-podinfo\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.908129 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.908102    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-istio-token\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.908223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.908181    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mldtp\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-kube-api-access-mldtp\") pod \"router-gateway-2-openshift-default-7fccdfc75c-sklct\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:48.989191 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:48.989127    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:49.466109 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:49.466073    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"]
Oct 30 19:30:49.475643 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:49.475605    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:49.475730 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:49.475691    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:49.475779 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:49.475731    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:30:50.017560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:50.017521    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" event={"ID":"bd99c339-6d14-4c35-856b-efebf44cc926","Type":"ContainerStarted","Data":"43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60"}
Oct 30 19:30:50.017965 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:50.017602    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" event={"ID":"bd99c339-6d14-4c35-856b-efebf44cc926","Type":"ContainerStarted","Data":"184cb650d2767b8b1bd1c36aee87198908f1e6c4d7110cae210c2c11ae2089ca"}
Oct 30 19:30:50.133132 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:50.133072    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podStartSLOduration=2.133057912 podStartE2EDuration="2.133057912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:50.131690496 +0000 UTC m=+3232.212276703" watchObservedRunningTime="2026-10-30 19:30:50.133057912 +0000 UTC m=+3232.213644118"
Oct 30 19:30:50.990070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:50.990036    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:50.994532 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:50.994507    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:51.023175 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:51.023146    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:51.023982 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:30:51.023963    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:30:58.104097 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:30:58.104060    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:31:00.407916 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.407874    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"]
Oct 30 19:31:00.412880 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.412855    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460259 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460224    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460259 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460261    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-dshm\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460280    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfd2\" (UniqueName: \"kubernetes.io/projected/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kube-api-access-8xfd2\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460362    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tmp-dir\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460394    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-model-cache\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460548 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460529    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5250a-1a95-4fb3-9645-e67fffc39d99-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460860 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460565    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tls-certs\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.460860 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.460592    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-home\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.464553 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.464528    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:31:00.512481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.511494    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-rk77l\""
Oct 30 19:31:00.512481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.511500    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Oct 30 19:31:00.512481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.511641    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Oct 30 19:31:00.512481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.511728    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-sn5g2\""
Oct 30 19:31:00.514626 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.514604    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"]
Oct 30 19:31:00.561443 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561397    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfd2\" (UniqueName: \"kubernetes.io/projected/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kube-api-access-8xfd2\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561472    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tmp-dir\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561496    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-model-cache\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561533    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5250a-1a95-4fb3-9645-e67fffc39d99-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561579    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tls-certs\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561625    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-home\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561918 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561739    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561918 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561766    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-dshm\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.561918 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561900    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tmp-dir\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.562074 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.561959    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-model-cache\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.562151 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.562129    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-home\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.562258 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.562239    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.562466 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.562441    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5250a-1a95-4fb3-9645-e67fffc39d99-cabundle-cert\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.564121 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.564100    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-dshm\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.564678 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.564654    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tls-certs\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.617782 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.617752    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfd2\" (UniqueName: \"kubernetes.io/projected/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kube-api-access-8xfd2\") pod \"router-with-refs-pd-test-kserve-699d5c4d65-lkkr5\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:00.731968 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:00.731878    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:01.179032 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:01.178987    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"]
Oct 30 19:31:01.184931 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:31:01.184903    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d5250a_1a95_4fb3_9645_e67fffc39d99.slice/crio-8789701f39685f165bd1153fea76100189ab9172eb0ca41ca072ef85059aeaa0 WatchSource:0}: Error finding container 8789701f39685f165bd1153fea76100189ab9172eb0ca41ca072ef85059aeaa0: Status 404 returned error can't find the container with id 8789701f39685f165bd1153fea76100189ab9172eb0ca41ca072ef85059aeaa0
Oct 30 19:31:02.099308 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:02.099268    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerStarted","Data":"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"}
Oct 30 19:31:02.099779 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:02.099352    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerStarted","Data":"8789701f39685f165bd1153fea76100189ab9172eb0ca41ca072ef85059aeaa0"}
Oct 30 19:31:02.099779 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:02.099523    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:03.108596 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:03.108567    5025 generic.go:356] "Generic (PLEG): container finished" podID="bd99c339-6d14-4c35-856b-efebf44cc926" containerID="43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60" exitCode=0
Oct 30 19:31:03.109044 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:03.108634    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" event={"ID":"bd99c339-6d14-4c35-856b-efebf44cc926","Type":"ContainerDied","Data":"43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60"}
Oct 30 19:31:03.109044 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:03.109037    5025 scope.go:122] "RemoveContainer" containerID="43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60"
Oct 30 19:31:03.110188 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:03.110156    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:03.110273 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:03.110231    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:03.110273 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:03.110270    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:31:03.110532 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:03.110514    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerStarted","Data":"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"}
Oct 30 19:31:04.120227 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:04.120156    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" event={"ID":"bd99c339-6d14-4c35-856b-efebf44cc926","Type":"ContainerStarted","Data":"20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044"}
Oct 30 19:31:04.989349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:04.989241    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:31:04.990495 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:04.990465    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:31:05.127195 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:05.127160    5025 generic.go:356] "Generic (PLEG): container finished" podID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerID="71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966" exitCode=0
Oct 30 19:31:05.127655 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:05.127232    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerDied","Data":"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"}
Oct 30 19:31:05.989785 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:05.989746    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:31:06.023877 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:06.023840    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:31:06.133810 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:06.133766    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerStarted","Data":"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"}
Oct 30 19:31:06.346438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:06.346309    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podStartSLOduration=3.86432853 podStartE2EDuration="6.346288256s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.481959726s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:06.344240664 +0000 UTC m=+3248.424826870" watchObservedRunningTime="2026-10-30 19:31:06.346288256 +0000 UTC m=+3248.426874464"
Oct 30 19:31:06.989925 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:06.989876    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:31:07.989876 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:07.989834    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:31:08.990090 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:08.990026    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 19:31:09.993948 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:09.993916    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:31:09.994968 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:09.994946    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:31:10.732384 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:10.732349    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:10.732555 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:10.732410    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:10.733777 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:10.733747    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:31:20.733077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:20.733027    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:31:20.744709 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:20.744684    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:31:28.089147 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:31:28.089105    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:31:30.732231 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:30.732185    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:31:40.732533 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:40.732432    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:31:50.732648 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:50.732599    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:31:58.082169 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:31:58.082128    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:31:58.489887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:58.489861    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:31:58.495984 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:31:58.495960    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:32:00.732932 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:00.732875    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:32:06.702652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.702613    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"]
Oct 30 19:32:06.706291 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.706269    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.791572 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.791520    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"]
Oct 30 19:32:06.806662 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.806630    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.806824 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.806671    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55wrs\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-kube-api-access-55wrs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.806824 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.806696    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.806950 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.806867    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.806950 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.806912    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.807063 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.806983    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.807119 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.807059    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecb2458-ba4f-4283-862d-e662e24aaa77-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.807119 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.807107    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.807226 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.807177    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908456 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908417    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908456 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908460    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908476    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908622    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecb2458-ba4f-4283-862d-e662e24aaa77-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908692    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908731    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908798    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908841    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55wrs\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-kube-api-access-55wrs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.908911 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.908882    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.909122 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.909068    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.909204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.909181    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.909467 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.909449    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.909560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.909486    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecb2458-ba4f-4283-862d-e662e24aaa77-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.909597 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.909553    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.911356 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.911335    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.911686 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.911662    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.965924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.965859    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:06.966131 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:06.966107    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55wrs\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-kube-api-access-55wrs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2vdhq\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:07.019278 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:07.019245    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:07.443202 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:07.443167    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"]
Oct 30 19:32:07.448783 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:32:07.448751    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ecb2458_ba4f_4283_862d_e662e24aaa77.slice/crio-bae61a5a4bc561b5e7b2aabe586c1df0615aba66d8b71dbe7c55dccd5b3d854c WatchSource:0}: Error finding container bae61a5a4bc561b5e7b2aabe586c1df0615aba66d8b71dbe7c55dccd5b3d854c: Status 404 returned error can't find the container with id bae61a5a4bc561b5e7b2aabe586c1df0615aba66d8b71dbe7c55dccd5b3d854c
Oct 30 19:32:07.450766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:07.450727    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:07.450878 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:07.450807    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:07.450878 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:07.450854    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:08.464352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:08.464226    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" event={"ID":"2ecb2458-ba4f-4283-862d-e662e24aaa77","Type":"ContainerStarted","Data":"975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1"}
Oct 30 19:32:08.464352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:08.464310    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" event={"ID":"2ecb2458-ba4f-4283-862d-e662e24aaa77","Type":"ContainerStarted","Data":"bae61a5a4bc561b5e7b2aabe586c1df0615aba66d8b71dbe7c55dccd5b3d854c"}
Oct 30 19:32:08.681344 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:08.678924    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podStartSLOduration=2.678903332 podStartE2EDuration="2.678903332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:08.628919667 +0000 UTC m=+3310.709505877" watchObservedRunningTime="2026-10-30 19:32:08.678903332 +0000 UTC m=+3310.759489542"
Oct 30 19:32:09.019472 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:09.019436    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:09.021200 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:09.021167    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:32:09.021364 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:09.021213    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:32:10.020392 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:10.020348    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:32:10.020791 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:10.020415    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:32:10.732982 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:10.732933    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:32:11.020558 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:11.020477    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:32:11.020558 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:11.020533    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:32:12.020343 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:12.020284    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:32:12.020518 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:12.020361    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:32:13.020418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:13.020380    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Oct 30 19:32:13.020835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:13.020440    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Oct 30 19:32:14.024904 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:14.024866    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:14.025420 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:14.025251    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:14.025899 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:14.025878    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:32:20.732912 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:20.732865    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:32:21.820812 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.820774    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"]
Oct 30 19:32:21.824476 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.824459    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973135 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973102    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973300 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973158    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973300 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973221    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973300 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973269    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973431 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973353    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973431 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973373    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxc4z\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-kube-api-access-rxc4z\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973431 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973393    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973431 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973421    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:21.973570 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:21.973505    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.015790 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.015757    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"]
Oct 30 19:32:22.074778 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.074686    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.074931 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.074784    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.074931 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.074827    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.074931 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.074864    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.074931 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.074897    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075147 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.074931    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxc4z\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-kube-api-access-rxc4z\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075147 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.074967    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075147 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.075011    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075147 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.075048    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.075242    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.075267    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075438 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.075340    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075676 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.075642    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.075833 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.075749    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.077566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.077540    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.077716 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.077699    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.131369 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.131339    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxc4z\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-kube-api-access-rxc4z\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.131842 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.131823    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-wjr8s\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.141523 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.141504    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:22.565428 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.565393    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"]
Oct 30 19:32:22.570688 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:32:22.570663    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce209e54_a1ce_4c93_8197_0aa0d9f62661.slice/crio-018d12e36fa3577911b157027d09e9df089fcabfcec7c6fc160c48b6e9578d8d WatchSource:0}: Error finding container 018d12e36fa3577911b157027d09e9df089fcabfcec7c6fc160c48b6e9578d8d: Status 404 returned error can't find the container with id 018d12e36fa3577911b157027d09e9df089fcabfcec7c6fc160c48b6e9578d8d
Oct 30 19:32:22.572874 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.572844    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:22.572963 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.572904    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:22.572963 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:22.572952    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:23.562346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:23.559840    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" event={"ID":"ce209e54-a1ce-4c93-8197-0aa0d9f62661","Type":"ContainerStarted","Data":"4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a"}
Oct 30 19:32:23.562346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:23.559917    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" event={"ID":"ce209e54-a1ce-4c93-8197-0aa0d9f62661","Type":"ContainerStarted","Data":"018d12e36fa3577911b157027d09e9df089fcabfcec7c6fc160c48b6e9578d8d"}
Oct 30 19:32:23.729076 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:23.729009    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podStartSLOduration=2.728994846 podStartE2EDuration="2.728994846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:23.726635296 +0000 UTC m=+3325.807221502" watchObservedRunningTime="2026-10-30 19:32:23.728994846 +0000 UTC m=+3325.809581054"
Oct 30 19:32:24.142685 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:24.142649    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:24.143891 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:24.143866    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:32:24.144002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:24.143915    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:32:25.142195 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:25.142153    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:32:25.142612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:25.142209    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:32:26.142166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:26.142130    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:32:26.142372 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:26.142183    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:32:27.142036 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:27.141998    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:32:27.142183 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:27.142081    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:32:28.103362 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:32:28.103308    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:32:28.142165 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:28.142127    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:32:28.142376 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:28.142196    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:32:29.146381 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:29.146348    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:29.146850 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:29.146611    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:29.147423 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:29.147404    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:32:30.732905 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:30.732856    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:32:36.938270 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:36.938233    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"]
Oct 30 19:32:36.942076 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:36.942059    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018124 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018081    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4xxx\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-kube-api-access-w4xxx\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018202    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018244    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018265    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018299    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576012b3-66fb-450a-86c0-0e2e5588f016-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018499 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018410    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018618 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018596    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018653 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018636    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.018697 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.018686    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576012b3-66fb-450a-86c0-0e2e5588f016-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.045007 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.044967    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"]
Oct 30 19:32:37.120129 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120090    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4xxx\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-kube-api-access-w4xxx\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120347 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120277    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120444 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120349    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120444 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120376    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120574 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120450    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576012b3-66fb-450a-86c0-0e2e5588f016-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120574 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120495    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120573    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120592    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120612    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576012b3-66fb-450a-86c0-0e2e5588f016-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120856 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120763    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.120856 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.120782    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.121064 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.121036    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.121465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.121248    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.121702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.121685    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576012b3-66fb-450a-86c0-0e2e5588f016-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.123255 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.123229    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.123528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.123509    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576012b3-66fb-450a-86c0-0e2e5588f016-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.176872 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.176836    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.177045 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.176933    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4xxx\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-kube-api-access-w4xxx\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-j7ndc\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.255831 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.255745    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:37.682221 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.682188    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"]
Oct 30 19:32:37.687990 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:32:37.687964    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576012b3_66fb_450a_86c0_0e2e5588f016.slice/crio-e1f97ea0626eead918304c95d1ea65f8bcf3d49c1aa736b4a969aff84ee8c7a0 WatchSource:0}: Error finding container e1f97ea0626eead918304c95d1ea65f8bcf3d49c1aa736b4a969aff84ee8c7a0: Status 404 returned error can't find the container with id e1f97ea0626eead918304c95d1ea65f8bcf3d49c1aa736b4a969aff84ee8c7a0
Oct 30 19:32:37.690205 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.690161    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:37.690287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.690255    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:37.690353 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:37.690309    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:32:38.656449 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:38.656410    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" event={"ID":"576012b3-66fb-450a-86c0-0e2e5588f016","Type":"ContainerStarted","Data":"2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde"}
Oct 30 19:32:38.656891 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:38.656470    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" event={"ID":"576012b3-66fb-450a-86c0-0e2e5588f016","Type":"ContainerStarted","Data":"e1f97ea0626eead918304c95d1ea65f8bcf3d49c1aa736b4a969aff84ee8c7a0"}
Oct 30 19:32:38.820424 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:38.820355    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podStartSLOduration=2.820314571 podStartE2EDuration="2.820314571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:32:38.818114334 +0000 UTC m=+3340.898700542" watchObservedRunningTime="2026-10-30 19:32:38.820314571 +0000 UTC m=+3340.900900778"
Oct 30 19:32:39.256933 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:39.256900    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:39.258191 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:39.258169    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:32:39.258263 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:39.258217    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:32:40.256451 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:40.256413    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:32:40.256853 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:40.256478    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:32:40.733235 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:40.733183    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:32:41.256289 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:41.256256    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:32:41.256453 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:41.256309    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:32:42.256584 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:42.256538    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:32:42.257006 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:42.256619    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:32:43.256700 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:43.256660    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 19:32:43.257207 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:43.256725    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 19:32:44.260259 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:44.260225    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:44.260798 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:44.260613    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:44.261256 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:44.261238    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:32:50.732381 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:32:50.732302    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8001/health\": dial tcp 10.132.0.88:8001: connect: connection refused"
Oct 30 19:32:58.081940 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:32:58.081908    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:33:00.742580 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:00.742548    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:33:00.754700 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:00.754677    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:33:13.569193 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:13.569113    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"]
Oct 30 19:33:28.082284 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:33:28.082248    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:33:28.593057 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:28.593022    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main" containerID="cri-o://7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b" gracePeriod=45
Oct 30 19:33:29.210816 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.210786    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="llm-d-routing-sidecar" containerID="cri-o://71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa" gracePeriod=45
Oct 30 19:33:29.464205 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.464175    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:33:29.549764 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.549728    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-dshm\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.549937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.549779    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tls-certs\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.549937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.549810    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kube-api-access-8xfd2\" (UniqueName: \"kubernetes.io/projected/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kube-api-access-8xfd2\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.549937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.549837    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-home\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.550075 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550000    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-model-cache\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.550075 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550063    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kserve-provision-location\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.550165 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550107    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d5250a-1a95-4fb3-9645-e67fffc39d99-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5250a-1a95-4fb3-9645-e67fffc39d99-cabundle-cert\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.550230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550193    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tmp-dir\") pod \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\" (UID: \"c2d5250a-1a95-4fb3-9645-e67fffc39d99\") "
Oct 30 19:33:29.550490 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550459    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-model-cache" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.550970 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550707    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-home" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.550970 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550839    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d5250a-1a95-4fb3-9645-e67fffc39d99-cabundle-cert" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:33:29.550970 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.550948    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.552443 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.552415    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kube-api-access-8xfd2" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "kube-api-access-8xfd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:33:29.552570 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.552436    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tls-certs" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:33:29.552570 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.552536    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-dshm" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.563459 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.563435    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tmp-dir" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.575213 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.575181    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kserve-provision-location" pod "c2d5250a-1a95-4fb3-9645-e67fffc39d99" (UID: "c2d5250a-1a95-4fb3-9645-e67fffc39d99"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:33:29.651884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.651800    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.651884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.651831    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c2d5250a-1a95-4fb3-9645-e67fffc39d99-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.651884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.651842    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.651884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.651851    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.651884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.651859    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2d5250a-1a95-4fb3-9645-e67fffc39d99-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.651884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.651867    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xfd2\" (UniqueName: \"kubernetes.io/projected/c2d5250a-1a95-4fb3-9645-e67fffc39d99-kube-api-access-8xfd2\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.651884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.651875    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2d5250a-1a95-4fb3-9645-e67fffc39d99-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:33:29.922604 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.922516    5025 generic.go:356] "Generic (PLEG): container finished" podID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerID="7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b" exitCode=0
Oct 30 19:33:29.922604 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.922542    5025 generic.go:356] "Generic (PLEG): container finished" podID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerID="71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa" exitCode=0
Oct 30 19:33:29.922604 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.922574    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerDied","Data":"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"}
Oct 30 19:33:29.922604 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.922595    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"
Oct 30 19:33:29.922854 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.922617    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerDied","Data":"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"}
Oct 30 19:33:29.922854 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.922645    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5" event={"ID":"c2d5250a-1a95-4fb3-9645-e67fffc39d99","Type":"ContainerDied","Data":"8789701f39685f165bd1153fea76100189ab9172eb0ca41ca072ef85059aeaa0"}
Oct 30 19:33:29.922854 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.922673    5025 scope.go:122] "RemoveContainer" containerID="7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"
Oct 30 19:33:29.937885 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.937862    5025 scope.go:122] "RemoveContainer" containerID="71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"
Oct 30 19:33:29.946475 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.946454    5025 scope.go:122] "RemoveContainer" containerID="71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"
Oct 30 19:33:29.954533 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.954513    5025 scope.go:122] "RemoveContainer" containerID="7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"
Oct 30 19:33:29.954909 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:33:29.954886    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b\": container with ID starting with 7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b not found: ID does not exist" containerID="7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"
Oct 30 19:33:29.955008 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.954919    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"} err="failed to get container status \"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b\": rpc error: code = NotFound desc = could not find container \"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b\": container with ID starting with 7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b not found: ID does not exist"
Oct 30 19:33:29.955008 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.954937    5025 scope.go:122] "RemoveContainer" containerID="71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"
Oct 30 19:33:29.955201 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:33:29.955184    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966\": container with ID starting with 71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966 not found: ID does not exist" containerID="71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"
Oct 30 19:33:29.955254 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.955205    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"} err="failed to get container status \"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966\": rpc error: code = NotFound desc = could not find container \"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966\": container with ID starting with 71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966 not found: ID does not exist"
Oct 30 19:33:29.955254 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.955230    5025 scope.go:122] "RemoveContainer" containerID="71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"
Oct 30 19:33:29.955510 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:33:29.955494    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa\": container with ID starting with 71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa not found: ID does not exist" containerID="71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"
Oct 30 19:33:29.955559 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.955513    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"} err="failed to get container status \"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa\": rpc error: code = NotFound desc = could not find container \"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa\": container with ID starting with 71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa not found: ID does not exist"
Oct 30 19:33:29.955559 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.955525    5025 scope.go:122] "RemoveContainer" containerID="7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"
Oct 30 19:33:29.955729 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.955713    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b"} err="failed to get container status \"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b\": rpc error: code = NotFound desc = could not find container \"7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b\": container with ID starting with 7048d225d87678ad362c54885f21d484a08f3733b972a18e96c2526fc4085e5b not found: ID does not exist"
Oct 30 19:33:29.955796 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.955729    5025 scope.go:122] "RemoveContainer" containerID="71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"
Oct 30 19:33:29.956003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.955982    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966"} err="failed to get container status \"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966\": rpc error: code = NotFound desc = could not find container \"71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966\": container with ID starting with 71cc6a3ca4400be27d005a623bad11ad2e89e25439a628be78462fbfe98b5966 not found: ID does not exist"
Oct 30 19:33:29.956058 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.956004    5025 scope.go:122] "RemoveContainer" containerID="71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"
Oct 30 19:33:29.956222 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:29.956206    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa"} err="failed to get container status \"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa\": rpc error: code = NotFound desc = could not find container \"71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa\": container with ID starting with 71d1f3ca66ba8b6281a671c1d6a372b3a78b2ee13d52382234c3814c795089aa not found: ID does not exist"
Oct 30 19:33:30.092042 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:30.090083    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"]
Oct 30 19:33:30.137107 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:30.137074    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-699d5c4d65-lkkr5"]
Oct 30 19:33:32.056663 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:33:32.056629    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" path="/var/lib/kubelet/pods/c2d5250a-1a95-4fb3-9645-e67fffc39d99/volumes"
Oct 30 19:33:58.084865 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:33:58.084833    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:34:08.037756 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:08.037720    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"]
Oct 30 19:34:08.038265 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:08.037914    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy" containerID="cri-o://20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044" gracePeriod=30
Oct 30 19:34:09.996529 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:09.996485    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:34:13.294731 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.294706    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:34:13.456773 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.456746    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd99c339-6d14-4c35-856b-efebf44cc926-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99c339-6d14-4c35-856b-efebf44cc926-istiod-ca-cert\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.456773 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.456781    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-certs\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.456805    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-socket\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.456826    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd99c339-6d14-4c35-856b-efebf44cc926-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99c339-6d14-4c35-856b-efebf44cc926-istio-podinfo\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.456896    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-envoy\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.456921    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-data\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.456966    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-credential-socket\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457022 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457021    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-kube-api-access-mldtp\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-kube-api-access-mldtp\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457354 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457047    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-istio-token\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-istio-token\") pod \"bd99c339-6d14-4c35-856b-efebf44cc926\" (UID: \"bd99c339-6d14-4c35-856b-efebf44cc926\") "
Oct 30 19:34:13.457354 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457070    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-certs" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.457354 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457166    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-socket" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.457354 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457285    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-data" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.457354 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457292    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd99c339-6d14-4c35-856b-efebf44cc926-istiod-ca-cert" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:34:13.457354 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457304    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-credential-socket" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.457633 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457429    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd99c339-6d14-4c35-856b-efebf44cc926-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.457633 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457444    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.457633 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457453    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.457633 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457461    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.457633 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.457476    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.459789 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.459762    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-envoy" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:34:13.459992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.459951    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-kube-api-access-mldtp" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "kube-api-access-mldtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:13.459992 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.459978    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd99c339-6d14-4c35-856b-efebf44cc926-istio-podinfo" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:34:13.460291 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.460272    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-istio-token" pod "bd99c339-6d14-4c35-856b-efebf44cc926" (UID: "bd99c339-6d14-4c35-856b-efebf44cc926"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:34:13.559043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.559007    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd99c339-6d14-4c35-856b-efebf44cc926-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.559194 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.559046    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mldtp\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-kube-api-access-mldtp\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.559194 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.559061    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd99c339-6d14-4c35-856b-efebf44cc926-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:13.559194 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:13.559075    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd99c339-6d14-4c35-856b-efebf44cc926-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:34:14.132798 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.132762    5025 generic.go:356] "Generic (PLEG): container finished" podID="bd99c339-6d14-4c35-856b-efebf44cc926" containerID="20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044" exitCode=0
Oct 30 19:34:14.132971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.132816    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"
Oct 30 19:34:14.132971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.132849    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" event={"ID":"bd99c339-6d14-4c35-856b-efebf44cc926","Type":"ContainerDied","Data":"20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044"}
Oct 30 19:34:14.132971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.132915    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct" event={"ID":"bd99c339-6d14-4c35-856b-efebf44cc926","Type":"ContainerDied","Data":"184cb650d2767b8b1bd1c36aee87198908f1e6c4d7110cae210c2c11ae2089ca"}
Oct 30 19:34:14.132971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.132944    5025 scope.go:122] "RemoveContainer" containerID="20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044"
Oct 30 19:34:14.144111 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.144072    5025 scope.go:122] "RemoveContainer" containerID="43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60"
Oct 30 19:34:14.152789 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.152771    5025 scope.go:122] "RemoveContainer" containerID="20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044"
Oct 30 19:34:14.153038 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:34:14.153019    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044\": container with ID starting with 20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044 not found: ID does not exist" containerID="20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044"
Oct 30 19:34:14.153083 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.153047    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044"} err="failed to get container status \"20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044\": rpc error: code = NotFound desc = could not find container \"20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044\": container with ID starting with 20ec4b9d9fa7fd4e6a59b73d8c9e69e16deafd5b7837a5bd21211d122eaf8044 not found: ID does not exist"
Oct 30 19:34:14.153083 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.153064    5025 scope.go:122] "RemoveContainer" containerID="43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60"
Oct 30 19:34:14.153290 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:34:14.153275    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60\": container with ID starting with 43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60 not found: ID does not exist" containerID="43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60"
Oct 30 19:34:14.153352 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.153294    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60"} err="failed to get container status \"43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60\": rpc error: code = NotFound desc = could not find container \"43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60\": container with ID starting with 43f95911684032dd1ffc4364d7b66d6114b21aa1c94f2c4c885c85673c8ebd60 not found: ID does not exist"
Oct 30 19:34:14.293299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.293261    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"]
Oct 30 19:34:14.299219 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:14.299191    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-gateway-2-openshift-default-7fccdfc75c-sklct"]
Oct 30 19:34:16.056465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:16.056434    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" path="/var/lib/kubelet/pods/bd99c339-6d14-4c35-856b-efebf44cc926/volumes"
Oct 30 19:34:21.551123 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551089    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"]
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551567    5025 cpu_manager.go:414] "containerMap: removing container" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy"
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551582    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy"
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551594    5025 cpu_manager.go:414] "containerMap: removing container" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy"
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551600    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy"
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551609    5025 cpu_manager.go:414] "containerMap: removing container" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="llm-d-routing-sidecar"
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551614    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="llm-d-routing-sidecar"
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551633    5025 cpu_manager.go:414] "containerMap: removing container" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="storage-initializer"
Oct 30 19:34:21.551638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551637    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="storage-initializer"
Oct 30 19:34:21.551924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551659    5025 cpu_manager.go:414] "containerMap: removing container" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main"
Oct 30 19:34:21.551924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551665    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main"
Oct 30 19:34:21.551924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551719    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="llm-d-routing-sidecar"
Oct 30 19:34:21.551924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551731    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d5250a-1a95-4fb3-9645-e67fffc39d99" containerName="main"
Oct 30 19:34:21.551924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551738    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy"
Oct 30 19:34:21.551924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.551753    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd99c339-6d14-4c35-856b-efebf44cc926" containerName="istio-proxy"
Oct 30 19:34:21.556145 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.556126    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.600282 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.600251    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"]
Oct 30 19:34:21.645644 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.645610    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"openshift-service-ca.crt\""
Oct 30 19:34:21.645830 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.645735    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"default-dockercfg-mplg4\""
Oct 30 19:34:21.645886 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.645825    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"kube-root-ca.crt\""
Oct 30 19:34:21.645886 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.645872    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-with-lora-adapters-a290d4b0\"/\"lora-pvc-adapters-test-kserve-self-signed-certs\""
Oct 30 19:34:21.734618 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.734585    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.734618 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.734617    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8ad146-8785-4486-85fb-3c06c8d974c0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.734875 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.734726    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.734875 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.734856    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.734985 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.734927    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.835638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.835539    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6bg5\" (UniqueName: \"kubernetes.io/projected/0b8ad146-8785-4486-85fb-3c06c8d974c0-kube-api-access-f6bg5\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.835638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.835590    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.835638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.835629    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.835926 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.835672    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.835926 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.835687    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8ad146-8785-4486-85fb-3c06c8d974c0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.835926 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.835730    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.836087 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.836063    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-model-cache\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.836652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.836606    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-tmp-dir\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.836771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.836654    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-home\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.839062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.839039    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-dshm\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.844292 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.844272    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8ad146-8785-4486-85fb-3c06c8d974c0-tls-certs\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.936217 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.936182    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6bg5\" (UniqueName: \"kubernetes.io/projected/0b8ad146-8785-4486-85fb-3c06c8d974c0-kube-api-access-f6bg5\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:21.992281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:21.992245    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6bg5\" (UniqueName: \"kubernetes.io/projected/0b8ad146-8785-4486-85fb-3c06c8d974c0-kube-api-access-f6bg5\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.170878 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.170838    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.273330 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.273298    5025 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") device path: \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.372530 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.372483    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.372712 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.372617    5025 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") DevicePath \"\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.422583 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.422504    5025 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") DevicePath \"csi-bea25c261025f6c9cd80e131fd51e3c945ef085c108f96d68042120cca7c6200\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.574561 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.574525    5025 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/85add36262e3a2c49235e99a6fa627f9b94bde9afc08b5a7e626acc11087f42d/globalmount\"" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.641158 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.641130    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") " pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:25.769856 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:25.769774    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:26.242483 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:26.242448    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"]
Oct 30 19:34:26.247724 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:34:26.247691    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8ad146_8785_4486_85fb_3c06c8d974c0.slice/crio-a114df129c1a2306d8e0bfb8c0334e27a69f931f18bcb5495004a1eba384e375 WatchSource:0}: Error finding container a114df129c1a2306d8e0bfb8c0334e27a69f931f18bcb5495004a1eba384e375: Status 404 returned error can't find the container with id a114df129c1a2306d8e0bfb8c0334e27a69f931f18bcb5495004a1eba384e375
Oct 30 19:34:27.195261 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:27.195224    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" event={"ID":"0b8ad146-8785-4486-85fb-3c06c8d974c0","Type":"ContainerStarted","Data":"d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153"}
Oct 30 19:34:27.195494 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:27.195276    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" event={"ID":"0b8ad146-8785-4486-85fb-3c06c8d974c0","Type":"ContainerStarted","Data":"a114df129c1a2306d8e0bfb8c0334e27a69f931f18bcb5495004a1eba384e375"}
Oct 30 19:34:27.404418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:27.404358    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podStartSLOduration=6.404343299 podStartE2EDuration="6.404343299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:27.403424842 +0000 UTC m=+3449.484011081" watchObservedRunningTime="2026-10-30 19:34:27.404343299 +0000 UTC m=+3449.484929503"
Oct 30 19:34:27.770380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:27.770337    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:28.082330 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:34:28.082226    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:34:35.770678 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:35.770638    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:34:35.772100 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:35.772072    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:34:45.778835 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:45.778742    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:34:55.771005 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:34:55.770958    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:34:58.082178 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:34:58.082134    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:35:05.770592 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:35:05.770537    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:35:15.770937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:35:15.770892    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:35:25.771184 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:35:25.771142    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:35:28.082346 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:35:28.082290    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:35:35.770562 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:35:35.770522    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.92:8000/health\": dial tcp 10.132.0.92:8000: connect: connection refused"
Oct 30 19:35:45.780356 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:35:45.780301    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:35:45.788156 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:35:45.788133    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:35:58.082504 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:35:58.082469    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:36:14.726903 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:14.726874    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-with-lora-adapters-a290d4b0_lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5_0b8ad146-8785-4486-85fb-3c06c8d974c0/main/0.log"
Oct 30 19:36:19.004491 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:19.004456    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"]
Oct 30 19:36:28.082176 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:36:28.082135    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:36:34.027534 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.027490    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main" containerID="cri-o://d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153" gracePeriod=45
Oct 30 19:36:34.690145 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.690114    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:36:34.738942 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.738874    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") pod \"0b8ad146-8785-4486-85fb-3c06c8d974c0\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") "
Oct 30 19:36:34.738942 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.738916    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-dshm\") pod \"0b8ad146-8785-4486-85fb-3c06c8d974c0\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") "
Oct 30 19:36:34.739171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.738944    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b8ad146-8785-4486-85fb-3c06c8d974c0-kube-api-access-f6bg5\" (UniqueName: \"kubernetes.io/projected/0b8ad146-8785-4486-85fb-3c06c8d974c0-kube-api-access-f6bg5\") pod \"0b8ad146-8785-4486-85fb-3c06c8d974c0\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") "
Oct 30 19:36:34.739171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.739045    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-tmp-dir\") pod \"0b8ad146-8785-4486-85fb-3c06c8d974c0\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") "
Oct 30 19:36:34.739171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.739070    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-home\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-home\") pod \"0b8ad146-8785-4486-85fb-3c06c8d974c0\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") "
Oct 30 19:36:34.739343 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.739163    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b8ad146-8785-4486-85fb-3c06c8d974c0-tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8ad146-8785-4486-85fb-3c06c8d974c0-tls-certs\") pod \"0b8ad146-8785-4486-85fb-3c06c8d974c0\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") "
Oct 30 19:36:34.739343 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.739225    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-model-cache\") pod \"0b8ad146-8785-4486-85fb-3c06c8d974c0\" (UID: \"0b8ad146-8785-4486-85fb-3c06c8d974c0\") "
Oct 30 19:36:34.739510 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.739490    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-home" pod "0b8ad146-8785-4486-85fb-3c06c8d974c0" (UID: "0b8ad146-8785-4486-85fb-3c06c8d974c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:34.739636 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.739622    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:34.739776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.739754    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-model-cache" pod "0b8ad146-8785-4486-85fb-3c06c8d974c0" (UID: "0b8ad146-8785-4486-85fb-3c06c8d974c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:34.741288 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.741266    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-dshm" pod "0b8ad146-8785-4486-85fb-3c06c8d974c0" (UID: "0b8ad146-8785-4486-85fb-3c06c8d974c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:34.741598 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.741577    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8ad146-8785-4486-85fb-3c06c8d974c0-kube-api-access-f6bg5" pod "0b8ad146-8785-4486-85fb-3c06c8d974c0" (UID: "0b8ad146-8785-4486-85fb-3c06c8d974c0"). InnerVolumeSpecName "kube-api-access-f6bg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:36:34.741719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.741695    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8ad146-8785-4486-85fb-3c06c8d974c0-tls-certs" pod "0b8ad146-8785-4486-85fb-3c06c8d974c0" (UID: "0b8ad146-8785-4486-85fb-3c06c8d974c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:36:34.748236 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.748108    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02" pod "0b8ad146-8785-4486-85fb-3c06c8d974c0" (UID: "0b8ad146-8785-4486-85fb-3c06c8d974c0"). InnerVolumeSpecName "pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Oct 30 19:36:34.750652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.750626    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-tmp-dir" pod "0b8ad146-8785-4486-85fb-3c06c8d974c0" (UID: "0b8ad146-8785-4486-85fb-3c06c8d974c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:36:34.814881 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.814849    5025 generic.go:356] "Generic (PLEG): container finished" podID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerID="d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153" exitCode=0
Oct 30 19:36:34.815047 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.814900    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"
Oct 30 19:36:34.815047 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.814930    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" event={"ID":"0b8ad146-8785-4486-85fb-3c06c8d974c0","Type":"ContainerDied","Data":"d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153"}
Oct 30 19:36:34.815047 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.814991    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5" event={"ID":"0b8ad146-8785-4486-85fb-3c06c8d974c0","Type":"ContainerDied","Data":"a114df129c1a2306d8e0bfb8c0334e27a69f931f18bcb5495004a1eba384e375"}
Oct 30 19:36:34.815047 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.815021    5025 scope.go:122] "RemoveContainer" containerID="d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153"
Oct 30 19:36:34.824385 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.824356    5025 scope.go:122] "RemoveContainer" containerID="d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153"
Oct 30 19:36:34.824645 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:36:34.824625    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153\": container with ID starting with d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153 not found: ID does not exist" containerID="d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153"
Oct 30 19:36:34.824702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.824653    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153"} err="failed to get container status \"d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153\": rpc error: code = NotFound desc = could not find container \"d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153\": container with ID starting with d694c5902f99f0731abaf8e01faeb5f1e2d592efe68791c89286b81bd0e12153 not found: ID does not exist"
Oct 30 19:36:34.840646 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.840622    5025 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" "
Oct 30 19:36:34.840744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.840647    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:34.840744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.840662    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6bg5\" (UniqueName: \"kubernetes.io/projected/0b8ad146-8785-4486-85fb-3c06c8d974c0-kube-api-access-f6bg5\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:34.840744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.840675    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:34.840744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.840688    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8ad146-8785-4486-85fb-3c06c8d974c0-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:34.840744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.840700    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8ad146-8785-4486-85fb-3c06c8d974c0-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:36:34.870961 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.870944    5025 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02") on node "ip-10-0-129-88.us-west-2.compute.internal"
Oct 30 19:36:34.941720 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:34.941690    5025 reconciler_common.go:299] "Volume detached for volume \"pvc-e292ab26-5e56-4ed2-bcae-958e9f1ede0a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0f53363343479af02\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"csi-bea25c261025f6c9cd80e131fd51e3c945ef085c108f96d68042120cca7c6200\""
Oct 30 19:36:35.009350 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:35.009255    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"]
Oct 30 19:36:35.014724 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:35.014701    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-with-lora-adapters-a290d4b0/lora-pvc-adapters-test-kserve-55d4fd88bc-gcww5"]
Oct 30 19:36:36.056271 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:36.056238    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" path="/var/lib/kubelet/pods/0b8ad146-8785-4486-85fb-3c06c8d974c0/volumes"
Oct 30 19:36:41.403481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.403445    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"]
Oct 30 19:36:41.404025 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.404004    5025 cpu_manager.go:414] "containerMap: removing container" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main"
Oct 30 19:36:41.404105 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.404027    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main"
Oct 30 19:36:41.404165 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.404154    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b8ad146-8785-4486-85fb-3c06c8d974c0" containerName="main"
Oct 30 19:36:41.408727 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.408707    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.460139 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.460107    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Oct 30 19:36:41.460139 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.460117    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Oct 30 19:36:41.497664 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.497634    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqvx4\" (UniqueName: \"kubernetes.io/projected/5e23a5c6-c8e7-490a-a7df-316328e4096e-kube-api-access-fqvx4\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.497828 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.497702    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e23a5c6-c8e7-490a-a7df-316328e4096e-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.507024 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.506999    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-xttbp\""
Oct 30 19:36:41.507024 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.507005    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Oct 30 19:36:41.507174 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.507036    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-5jc6n\""
Oct 30 19:36:41.509228 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.509208    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"]
Oct 30 19:36:41.593430 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.593399    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 19:36:41.596995 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.596978    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.598289 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.598266    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqvx4\" (UniqueName: \"kubernetes.io/projected/5e23a5c6-c8e7-490a-a7df-316328e4096e-kube-api-access-fqvx4\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.598399 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.598359    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e23a5c6-c8e7-490a-a7df-316328e4096e-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.600817 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.600791    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e23a5c6-c8e7-490a-a7df-316328e4096e-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.654638 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.654562    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 19:36:41.659533 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.659503    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqvx4\" (UniqueName: \"kubernetes.io/projected/5e23a5c6-c8e7-490a-a7df-316328e4096e-kube-api-access-fqvx4\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.660457 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.660439    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:36:41.660727 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.660709    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Oct 30 19:36:41.661025 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.661009    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-j99zj\""
Oct 30 19:36:41.699597 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.699569    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869qb\" (UniqueName: \"kubernetes.io/projected/7241f51b-3078-4f88-b21f-8bc14c14afb1-kube-api-access-869qb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.699752 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.699629    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7241f51b-3078-4f88-b21f-8bc14c14afb1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.699752 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.699732    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.699843 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.699769    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.699898 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.699879    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.699949 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.699910    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.700019 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.700000    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7241f51b-3078-4f88-b21f-8bc14c14afb1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.700077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.700048    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.721904 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.721878    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:41.801206 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801175    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7241f51b-3078-4f88-b21f-8bc14c14afb1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.801308 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801221    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.801308 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801242    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.801308 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801281    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.801308 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801299    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.801469 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801343    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7241f51b-3078-4f88-b21f-8bc14c14afb1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.801469 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801363    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.801469 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.801384    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869qb\" (UniqueName: \"kubernetes.io/projected/7241f51b-3078-4f88-b21f-8bc14c14afb1-kube-api-access-869qb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.802109 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.802089    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7241f51b-3078-4f88-b21f-8bc14c14afb1-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.802286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.802266    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.802795 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.802770    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.802893 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.802870    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.803105 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.803086    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.804960 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.804940    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.805654 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.805635    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7241f51b-3078-4f88-b21f-8bc14c14afb1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.857411 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.857385    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-869qb\" (UniqueName: \"kubernetes.io/projected/7241f51b-3078-4f88-b21f-8bc14c14afb1-kube-api-access-869qb\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:41.920485 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:41.920411    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:36:42.097950 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.097917    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"]
Oct 30 19:36:42.103472 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:36:42.103446    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e23a5c6_c8e7_490a_a7df_316328e4096e.slice/crio-490bb91c1c4f928e79bb9f881501a9081689ee438ab451d1fef60f0915c949f8 WatchSource:0}: Error finding container 490bb91c1c4f928e79bb9f881501a9081689ee438ab451d1fef60f0915c949f8: Status 404 returned error can't find the container with id 490bb91c1c4f928e79bb9f881501a9081689ee438ab451d1fef60f0915c949f8
Oct 30 19:36:42.294848 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.294762    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 19:36:42.300167 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:36:42.300136    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7241f51b_3078_4f88_b21f_8bc14c14afb1.slice/crio-df501f47cc29cda9b15dcb5a3e1488903c3f6683126b1d114903132c6c76a1ee WatchSource:0}: Error finding container df501f47cc29cda9b15dcb5a3e1488903c3f6683126b1d114903132c6c76a1ee: Status 404 returned error can't find the container with id df501f47cc29cda9b15dcb5a3e1488903c3f6683126b1d114903132c6c76a1ee
Oct 30 19:36:42.854231 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.854193    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" event={"ID":"5e23a5c6-c8e7-490a-a7df-316328e4096e","Type":"ContainerStarted","Data":"fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32"}
Oct 30 19:36:42.854723 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.854247    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" event={"ID":"5e23a5c6-c8e7-490a-a7df-316328e4096e","Type":"ContainerStarted","Data":"490bb91c1c4f928e79bb9f881501a9081689ee438ab451d1fef60f0915c949f8"}
Oct 30 19:36:42.854723 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.854277    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:36:42.855852 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.855828    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7241f51b-3078-4f88-b21f-8bc14c14afb1","Type":"ContainerStarted","Data":"3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81"}
Oct 30 19:36:42.855961 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.855870    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7241f51b-3078-4f88-b21f-8bc14c14afb1","Type":"ContainerStarted","Data":"df501f47cc29cda9b15dcb5a3e1488903c3f6683126b1d114903132c6c76a1ee"}
Oct 30 19:36:42.965581 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:42.965531    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" podStartSLOduration=1.9655171089999999 podStartE2EDuration="1.965517109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:42.964230852 +0000 UTC m=+3585.044817059" watchObservedRunningTime="2026-10-30 19:36:42.965517109 +0000 UTC m=+3585.046103315"
Oct 30 19:36:44.871747 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:44.871707    5025 generic.go:356] "Generic (PLEG): container finished" podID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerID="3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81" exitCode=0
Oct 30 19:36:44.872157 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:44.871778    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7241f51b-3078-4f88-b21f-8bc14c14afb1","Type":"ContainerDied","Data":"3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81"}
Oct 30 19:36:45.879623 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:45.879589    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7241f51b-3078-4f88-b21f-8bc14c14afb1","Type":"ContainerStarted","Data":"94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de"}
Oct 30 19:36:58.090838 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:36:58.090791    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:36:58.522971 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:58.522943    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:36:58.530930 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:36:58.530905    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:37:13.868985 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:13.868953    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:37:13.981024 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:13.980961    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=30.713373728 podStartE2EDuration="32.980945856s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.267572128s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:46.088886598 +0000 UTC m=+3588.169472805" watchObservedRunningTime="2026-10-30 19:37:13.980945856 +0000 UTC m=+3616.061532063"
Oct 30 19:37:24.077192 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.077156    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"]
Oct 30 19:37:24.077673 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.077366    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy" containerID="cri-o://c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5" gracePeriod=30
Oct 30 19:37:24.124696 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.124663    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"]
Oct 30 19:37:24.124876 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.124856    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy" containerID="cri-o://2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde" gracePeriod=30
Oct 30 19:37:24.129566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.128335    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"]
Oct 30 19:37:24.129566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.128773    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy" containerID="cri-o://4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a" gracePeriod=30
Oct 30 19:37:24.172917 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.172881    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"]
Oct 30 19:37:24.173394 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:24.173337    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy" containerID="cri-o://975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1" gracePeriod=30
Oct 30 19:37:25.501245 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:25.501209    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:37:25.501639 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:25.501265    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:37:28.082314 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:37:28.082276    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:37:29.027756 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.027714    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 19:37:29.027919 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.027793    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 19:37:29.147807 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.147773    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 19:37:29.148282 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.147830    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 19:37:29.379448 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.376982    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:37:29.382049 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.381968    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:37:29.416922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.416894    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 19:37:29.462643 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462613    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-envoy\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.462831 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462657    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-credential-socket\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.462831 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462686    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576012b3-66fb-450a-86c0-0e2e5588f016-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576012b3-66fb-450a-86c0-0e2e5588f016-istiod-ca-cert\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.462831 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462722    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-podinfo\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.462831 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462778    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-certs\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.463059 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462887    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-kube-api-access-w4xxx\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-kube-api-access-w4xxx\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.463059 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462935    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-envoy\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.463059 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462962    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istiod-ca-cert\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.463059 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.462991    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-data\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.463059 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463035    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/576012b3-66fb-450a-86c0-0e2e5588f016-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576012b3-66fb-450a-86c0-0e2e5588f016-istio-podinfo\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.463059 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463059    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-data\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463060    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-credential-socket" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463095    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-credential-socket\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463132    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-socket\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463141    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-certs" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463169    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-socket\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463190    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576012b3-66fb-450a-86c0-0e2e5588f016-istiod-ca-cert" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463214    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-credential-socket\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463244    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-certs\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463272    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-envoy\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463336    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-socket\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463355    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-data" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463365    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-kube-api-access-2tr8r\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-kube-api-access-2tr8r\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.463404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463395    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-data\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463428    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-kube-api-access-rxc4z\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-kube-api-access-rxc4z\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463440    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istiod-ca-cert" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463468    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istiod-ca-cert\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463534    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-certs\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463561    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-data" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463563    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-podinfo\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463607    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-credential-socket" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463865    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-socket" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463978    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-istio-token\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-istio-token\") pod \"576012b3-66fb-450a-86c0-0e2e5588f016\" (UID: \"576012b3-66fb-450a-86c0-0e2e5588f016\") "
Oct 30 19:37:29.464067 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.464046    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-token\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-token\") pod \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\" (UID: \"ce209e54-a1ce-4c93-8197-0aa0d9f62661\") "
Oct 30 19:37:29.464652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.464087    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-token\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-token\") pod \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\" (UID: \"64168401-3f30-4ea0-939c-5f2ba1cab7f1\") "
Oct 30 19:37:29.464652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.463978    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-credential-socket" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.464652 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.464447    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-socket" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.465205 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.465101    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-certs" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.465444 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.465347    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-data" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.465444 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.465358    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-certs" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.465725 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.464158    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-socket" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.466499 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.466379    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istiod-ca-cert" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:29.466702 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.466475    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.466893 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.466879    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576012b3-66fb-450a-86c0-0e2e5588f016-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.467009 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.466997    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.467105 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467092    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.467513 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467499    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.467637 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467624    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.467756 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467743    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.467966 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467953    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.468076 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468064    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.468239 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468211    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.468381 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468366    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.468455 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468406    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.468455 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468437    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.468560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467053    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-podinfo" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:29.468560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467097    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-envoy" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.468560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.467875    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-podinfo" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:29.468560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468094    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-istio-token" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.468560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468147    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-kube-api-access-rxc4z" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "kube-api-access-rxc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.468560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468249    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/576012b3-66fb-450a-86c0-0e2e5588f016-istio-podinfo" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:29.468560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468260    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-kube-api-access-w4xxx" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "kube-api-access-w4xxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.468817 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468565    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.468817 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.468583    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-envoy" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.469560 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.469538    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-token" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.469894 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.469871    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-envoy" pod "576012b3-66fb-450a-86c0-0e2e5588f016" (UID: "576012b3-66fb-450a-86c0-0e2e5588f016"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.470005 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.469967    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-token" pod "ce209e54-a1ce-4c93-8197-0aa0d9f62661" (UID: "ce209e54-a1ce-4c93-8197-0aa0d9f62661"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.470930 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.470910    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-kube-api-access-2tr8r" pod "64168401-3f30-4ea0-939c-5f2ba1cab7f1" (UID: "64168401-3f30-4ea0-939c-5f2ba1cab7f1"). InnerVolumeSpecName "kube-api-access-2tr8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.481128 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.481103    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:37:29.569902 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.569809    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-token\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-token\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.569902 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.569896    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-certs\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570141 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.569954    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-envoy\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570141 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.569980    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-socket\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570141 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570034    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ecb2458-ba4f-4283-862d-e662e24aaa77-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecb2458-ba4f-4283-862d-e662e24aaa77-istiod-ca-cert\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570141 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570074    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-podinfo\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570385 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570255    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-certs" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.570385 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570277    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-socket" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.570491 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570453    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-credential-socket\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570547 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570495    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-data\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570601 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570541    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-kube-api-access-55wrs\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-kube-api-access-55wrs\") pod \"2ecb2458-ba4f-4283-862d-e662e24aaa77\" (UID: \"2ecb2458-ba4f-4283-862d-e662e24aaa77\") "
Oct 30 19:37:29.570780 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570662    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ecb2458-ba4f-4283-862d-e662e24aaa77-istiod-ca-cert" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:37:29.570864 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570758    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-credential-socket" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.570920 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570901    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.570978 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570922    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.570978 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570925    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-data" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.570978 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570937    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.570978 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570952    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.570978 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570967    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ecb2458-ba4f-4283-862d-e662e24aaa77-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570982    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.570997    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571011    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571025    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4xxx\" (UniqueName: \"kubernetes.io/projected/576012b3-66fb-450a-86c0-0e2e5588f016-kube-api-access-w4xxx\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571040    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64168401-3f30-4ea0-939c-5f2ba1cab7f1-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571055    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576012b3-66fb-450a-86c0-0e2e5588f016-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571069    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576012b3-66fb-450a-86c0-0e2e5588f016-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571081    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tr8r\" (UniqueName: \"kubernetes.io/projected/64168401-3f30-4ea0-939c-5f2ba1cab7f1-kube-api-access-2tr8r\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571095    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571109    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxc4z\" (UniqueName: \"kubernetes.io/projected/ce209e54-a1ce-4c93-8197-0aa0d9f62661-kube-api-access-rxc4z\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.571281 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.571122    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce209e54-a1ce-4c93-8197-0aa0d9f62661-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.572463 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.572437    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-envoy" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:37:29.573003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.572979    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-token" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.573133 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.573109    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-podinfo" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:37:29.573191 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.573123    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-kube-api-access-55wrs" pod "2ecb2458-ba4f-4283-862d-e662e24aaa77" (UID: "2ecb2458-ba4f-4283-862d-e662e24aaa77"). InnerVolumeSpecName "kube-api-access-55wrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:37:29.671809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.671773    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.671809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.671805    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.671809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.671814    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55wrs\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-kube-api-access-55wrs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.672043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.671823    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.672043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.671832    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:29.672043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:29.671840    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ecb2458-ba4f-4283-862d-e662e24aaa77-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:37:30.101952 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.101915    5025 generic.go:356] "Generic (PLEG): container finished" podID="576012b3-66fb-450a-86c0-0e2e5588f016" containerID="2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde" exitCode=0
Oct 30 19:37:30.102085 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.101995    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"
Oct 30 19:37:30.102155 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.101992    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" event={"ID":"576012b3-66fb-450a-86c0-0e2e5588f016","Type":"ContainerDied","Data":"2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde"}
Oct 30 19:37:30.102213 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.102188    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" event={"ID":"576012b3-66fb-450a-86c0-0e2e5588f016","Type":"ContainerDied","Data":"e1f97ea0626eead918304c95d1ea65f8bcf3d49c1aa736b4a969aff84ee8c7a0"}
Oct 30 19:37:30.102268 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.102225    5025 scope.go:122] "RemoveContainer" containerID="2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde"
Oct 30 19:37:30.103620 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.103597    5025 generic.go:356] "Generic (PLEG): container finished" podID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerID="c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5" exitCode=0
Oct 30 19:37:30.103733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.103662    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"
Oct 30 19:37:30.103733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.103677    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" event={"ID":"64168401-3f30-4ea0-939c-5f2ba1cab7f1","Type":"ContainerDied","Data":"c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5"}
Oct 30 19:37:30.103733 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.103723    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl" event={"ID":"64168401-3f30-4ea0-939c-5f2ba1cab7f1","Type":"ContainerDied","Data":"9f8e2739b067f2ea3cc734dc07f882593ab3f23f65b02e66bc981f0c43f5019e"}
Oct 30 19:37:30.105363 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.105339    5025 generic.go:356] "Generic (PLEG): container finished" podID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerID="975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1" exitCode=0
Oct 30 19:37:30.105481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.105386    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"
Oct 30 19:37:30.105481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.105410    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" event={"ID":"2ecb2458-ba4f-4283-862d-e662e24aaa77","Type":"ContainerDied","Data":"975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1"}
Oct 30 19:37:30.105481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.105466    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq" event={"ID":"2ecb2458-ba4f-4283-862d-e662e24aaa77","Type":"ContainerDied","Data":"bae61a5a4bc561b5e7b2aabe586c1df0615aba66d8b71dbe7c55dccd5b3d854c"}
Oct 30 19:37:30.107301 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.107281    5025 generic.go:356] "Generic (PLEG): container finished" podID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerID="4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a" exitCode=0
Oct 30 19:37:30.107376 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.107309    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" event={"ID":"ce209e54-a1ce-4c93-8197-0aa0d9f62661","Type":"ContainerDied","Data":"4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a"}
Oct 30 19:37:30.107376 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.107363    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s" event={"ID":"ce209e54-a1ce-4c93-8197-0aa0d9f62661","Type":"ContainerDied","Data":"018d12e36fa3577911b157027d09e9df089fcabfcec7c6fc160c48b6e9578d8d"}
Oct 30 19:37:30.107525 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.107363    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"
Oct 30 19:37:30.131046 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.131024    5025 scope.go:122] "RemoveContainer" containerID="2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde"
Oct 30 19:37:30.131408 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:37:30.131385    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde\": container with ID starting with 2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde not found: ID does not exist" containerID="2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde"
Oct 30 19:37:30.131502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.131420    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde"} err="failed to get container status \"2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde\": rpc error: code = NotFound desc = could not find container \"2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde\": container with ID starting with 2c6d1702ea40e46d28bed919e95353e74e951fe07c649bbab4ccf4dbecc36dde not found: ID does not exist"
Oct 30 19:37:30.131502 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.131444    5025 scope.go:122] "RemoveContainer" containerID="c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5"
Oct 30 19:37:30.139865 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.139841    5025 scope.go:122] "RemoveContainer" containerID="c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5"
Oct 30 19:37:30.140121 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:37:30.140104    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5\": container with ID starting with c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5 not found: ID does not exist" containerID="c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5"
Oct 30 19:37:30.140179 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.140128    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5"} err="failed to get container status \"c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5\": rpc error: code = NotFound desc = could not find container \"c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5\": container with ID starting with c90800b9f4ec7c958572ab45ce2bffb83aeaae80b72a53398be9556ba140f4c5 not found: ID does not exist"
Oct 30 19:37:30.140179 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.140146    5025 scope.go:122] "RemoveContainer" containerID="975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1"
Oct 30 19:37:30.148186 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.148164    5025 scope.go:122] "RemoveContainer" containerID="975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1"
Oct 30 19:37:30.148510 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:37:30.148480    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1\": container with ID starting with 975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1 not found: ID does not exist" containerID="975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1"
Oct 30 19:37:30.148563 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.148507    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1"} err="failed to get container status \"975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1\": rpc error: code = NotFound desc = could not find container \"975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1\": container with ID starting with 975be162aa5f9b7c63abd139f8895804ac27ed1f633cfcae5adb5746e5f424d1 not found: ID does not exist"
Oct 30 19:37:30.148563 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.148526    5025 scope.go:122] "RemoveContainer" containerID="4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a"
Oct 30 19:37:30.156923 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.156904    5025 scope.go:122] "RemoveContainer" containerID="4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a"
Oct 30 19:37:30.157256 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:37:30.157239    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a\": container with ID starting with 4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a not found: ID does not exist" containerID="4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a"
Oct 30 19:37:30.157302 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.157267    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a"} err="failed to get container status \"4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a\": rpc error: code = NotFound desc = could not find container \"4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a\": container with ID starting with 4bc14613d91ede29e03b05cb934361d788f1087eca78535d224654694019e93a not found: ID does not exist"
Oct 30 19:37:30.260935 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.260901    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 19:37:30.261118 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.260956    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 19:37:30.261904 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.261877    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"]
Oct 30 19:37:30.266552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.266521    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-fcdrl"]
Oct 30 19:37:30.372196 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.372163    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"]
Oct 30 19:37:30.377986 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.377958    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2vdhq"]
Oct 30 19:37:30.483981 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.483940    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"]
Oct 30 19:37:30.491088 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.491058    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-wjr8s"]
Oct 30 19:37:30.596647 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.596610    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"]
Oct 30 19:37:30.602149 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:30.602122    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-j7ndc"]
Oct 30 19:37:32.058365 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:32.058314    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" path="/var/lib/kubelet/pods/2ecb2458-ba4f-4283-862d-e662e24aaa77/volumes"
Oct 30 19:37:32.058847 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:32.058758    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" path="/var/lib/kubelet/pods/576012b3-66fb-450a-86c0-0e2e5588f016/volumes"
Oct 30 19:37:32.059145 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:32.059129    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" path="/var/lib/kubelet/pods/64168401-3f30-4ea0-939c-5f2ba1cab7f1/volumes"
Oct 30 19:37:32.059891 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:37:32.059872    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" path="/var/lib/kubelet/pods/ce209e54-a1ce-4c93-8197-0aa0d9f62661/volumes"
Oct 30 19:37:58.082293 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:37:58.082260    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:38:04.224754 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.224716    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"]
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225083    5025 cpu_manager.go:414] "containerMap: removing container" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy"
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225094    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy"
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225104    5025 cpu_manager.go:414] "containerMap: removing container" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy"
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225109    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy"
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225128    5025 cpu_manager.go:414] "containerMap: removing container" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy"
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225133    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy"
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225141    5025 cpu_manager.go:414] "containerMap: removing container" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy"
Oct 30 19:38:04.225166 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225146    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy"
Oct 30 19:38:04.225446 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225208    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="576012b3-66fb-450a-86c0-0e2e5588f016" containerName="istio-proxy"
Oct 30 19:38:04.225446 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225217    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="64168401-3f30-4ea0-939c-5f2ba1cab7f1" containerName="istio-proxy"
Oct 30 19:38:04.225446 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225226    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce209e54-a1ce-4c93-8197-0aa0d9f62661" containerName="istio-proxy"
Oct 30 19:38:04.225446 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.225232    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ecb2458-ba4f-4283-862d-e662e24aaa77" containerName="istio-proxy"
Oct 30 19:38:04.229291 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.229270    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.280923 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.280889    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:38:04.314572 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.314546    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"kube-root-ca.crt\""
Oct 30 19:38:04.314572 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.314562    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"openshift-service-ca.crt\""
Oct 30 19:38:04.326949 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.326926    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-based-routing-only-fae765e6\"/\"model-routing-only-gateway-openshift-default-dockercfg-kcqhv\""
Oct 30 19:38:04.328813 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.328790    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"]
Oct 30 19:38:04.385675 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385647    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.385817 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385692    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.385817 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385781    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.385935 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385822    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.385935 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385850    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6tg2\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-kube-api-access-k6tg2\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.386003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385951    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.386003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385971    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.386003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.385995    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.386153 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.386020    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487193 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487099    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487193 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487147    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487193 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487175    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487205    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487243    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487267    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487293    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6tg2\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-kube-api-access-k6tg2\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487363    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487528 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487390    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487843 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487764    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487907 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487851    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-data\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.487966 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487954    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-certs\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.488024 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.487955    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-credential-socket\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.488348 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.488303    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istiod-ca-cert\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.489887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.489866    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-podinfo\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.490048 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.490028    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-envoy\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.543845 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.543812    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-token\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.544228 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.544205    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6tg2\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-kube-api-access-k6tg2\") pod \"model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") " pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:04.843620 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:04.843536    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:05.273824 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:05.273775    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"]
Oct 30 19:38:05.280504 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:38:05.280478    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7abe904_2a3d_4bad_8b33_e2f4b62af417.slice/crio-e7cf53a4cd19c15a7ea28aac473c810e4f1400e8d754077a2b7479e37b6a8989 WatchSource:0}: Error finding container e7cf53a4cd19c15a7ea28aac473c810e4f1400e8d754077a2b7479e37b6a8989: Status 404 returned error can't find the container with id e7cf53a4cd19c15a7ea28aac473c810e4f1400e8d754077a2b7479e37b6a8989
Oct 30 19:38:05.282473 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:05.282435    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:38:05.282562 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:05.282515    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:38:05.282562 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:05.282543    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:38:06.285806 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:06.285768    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" event={"ID":"f7abe904-2a3d-4bad-8b33-e2f4b62af417","Type":"ContainerStarted","Data":"a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064"}
Oct 30 19:38:06.286215 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:06.285830    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" event={"ID":"f7abe904-2a3d-4bad-8b33-e2f4b62af417","Type":"ContainerStarted","Data":"e7cf53a4cd19c15a7ea28aac473c810e4f1400e8d754077a2b7479e37b6a8989"}
Oct 30 19:38:06.497721 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:06.497663    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" podStartSLOduration=2.497648451 podStartE2EDuration="2.497648451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:38:06.449637407 +0000 UTC m=+3668.530223646" watchObservedRunningTime="2026-10-30 19:38:06.497648451 +0000 UTC m=+3668.578234636"
Oct 30 19:38:06.844124 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:06.844088    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:06.849010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:06.848982    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:07.290239 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:07.290197    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:07.291354 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:07.291309    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:28.082469 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:38:28.082427    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:38:29.415973 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:29.415943    5025 generic.go:356] "Generic (PLEG): container finished" podID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerID="a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064" exitCode=0
Oct 30 19:38:29.416470 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:29.416018    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" event={"ID":"f7abe904-2a3d-4bad-8b33-e2f4b62af417","Type":"ContainerDied","Data":"a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064"}
Oct 30 19:38:29.416470 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:29.416455    5025 scope.go:122] "RemoveContainer" containerID="a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064"
Oct 30 19:38:29.417449 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:29.417422    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:38:29.417508 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:29.417489    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:38:29.417546 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:29.417524    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:38:30.422046 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:30.422015    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" event={"ID":"f7abe904-2a3d-4bad-8b33-e2f4b62af417","Type":"ContainerStarted","Data":"ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83"}
Oct 30 19:38:30.844634 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:30.844546    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:30.844809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:30.844782    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 19:38:31.256448 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:31.256415    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"]
Oct 30 19:38:31.844481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:31.844443    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 19:38:32.844747 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:32.844702    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 19:38:33.844148 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:33.844103    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 19:38:33.884197 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:33.884155    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:38:34.844180 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:34.844141    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 19:38:35.848487 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:35.848454    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:35.848987 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:35.848834    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:35.849423 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:35.849403    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:38:43.869028 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:43.868981    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:38:46.303445 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:46.303389    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main" containerID="cri-o://fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32" gracePeriod=45
Oct 30 19:38:48.208098 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:48.208061    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 19:38:53.869101 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:53.869062    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:38:53.869518 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:38:53.869180    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:38:58.081869 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:38:58.081833    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:39:03.230865 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:03.230826    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerName="main" containerID="cri-o://94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de" gracePeriod=45
Oct 30 19:39:03.869360 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:03.869298    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:03.977077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:03.977051    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:39:04.092160 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092083    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7241f51b-3078-4f88-b21f-8bc14c14afb1-kube-api-access-869qb\" (UniqueName: \"kubernetes.io/projected/7241f51b-3078-4f88-b21f-8bc14c14afb1-kube-api-access-869qb\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092160 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092153    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-home\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-home\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092205    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7241f51b-3078-4f88-b21f-8bc14c14afb1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7241f51b-3078-4f88-b21f-8bc14c14afb1-cabundle-cert\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092228    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-kserve-provision-location\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092246    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-dshm\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092274    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-model-cache\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092351    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241f51b-3078-4f88-b21f-8bc14c14afb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7241f51b-3078-4f88-b21f-8bc14c14afb1-tls-certs\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092386    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-tmp-dir\") pod \"7241f51b-3078-4f88-b21f-8bc14c14afb1\" (UID: \"7241f51b-3078-4f88-b21f-8bc14c14afb1\") "
Oct 30 19:39:04.092737 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092633    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7241f51b-3078-4f88-b21f-8bc14c14afb1-cabundle-cert" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:04.092977 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.092952    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-home" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:04.093142 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.093120    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-model-cache" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:04.094646 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.094621    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-dshm" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:04.094811 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.094792    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7241f51b-3078-4f88-b21f-8bc14c14afb1-kube-api-access-869qb" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "kube-api-access-869qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:04.095034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.095018    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241f51b-3078-4f88-b21f-8bc14c14afb1-tls-certs" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:04.106135 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.106105    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-tmp-dir" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:04.116457 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.116414    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-kserve-provision-location" pod "7241f51b-3078-4f88-b21f-8bc14c14afb1" (UID: "7241f51b-3078-4f88-b21f-8bc14c14afb1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:04.193614 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193582    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869qb\" (UniqueName: \"kubernetes.io/projected/7241f51b-3078-4f88-b21f-8bc14c14afb1-kube-api-access-869qb\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.193614 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193608    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.193614 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193618    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7241f51b-3078-4f88-b21f-8bc14c14afb1-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.193830 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193627    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.193830 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193637    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.193830 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193644    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.193830 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193654    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7241f51b-3078-4f88-b21f-8bc14c14afb1-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.193830 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.193662    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7241f51b-3078-4f88-b21f-8bc14c14afb1-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:04.585428 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.585387    5025 generic.go:356] "Generic (PLEG): container finished" podID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerID="94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de" exitCode=0
Oct 30 19:39:04.585903 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.585441    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Oct 30 19:39:04.585903 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.585452    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7241f51b-3078-4f88-b21f-8bc14c14afb1","Type":"ContainerDied","Data":"94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de"}
Oct 30 19:39:04.585903 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.585506    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"7241f51b-3078-4f88-b21f-8bc14c14afb1","Type":"ContainerDied","Data":"df501f47cc29cda9b15dcb5a3e1488903c3f6683126b1d114903132c6c76a1ee"}
Oct 30 19:39:04.585903 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.585539    5025 scope.go:122] "RemoveContainer" containerID="94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de"
Oct 30 19:39:04.595256 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.595238    5025 scope.go:122] "RemoveContainer" containerID="3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81"
Oct 30 19:39:04.602943 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.602924    5025 scope.go:122] "RemoveContainer" containerID="94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de"
Oct 30 19:39:04.603188 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:39:04.603169    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de\": container with ID starting with 94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de not found: ID does not exist" containerID="94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de"
Oct 30 19:39:04.603234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.603196    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de"} err="failed to get container status \"94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de\": rpc error: code = NotFound desc = could not find container \"94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de\": container with ID starting with 94630dbf192cc39f33de29fcc57a220b7c0009c9610f6efd025ec4edfc0c85de not found: ID does not exist"
Oct 30 19:39:04.603234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.603214    5025 scope.go:122] "RemoveContainer" containerID="3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81"
Oct 30 19:39:04.603429 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:39:04.603414    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81\": container with ID starting with 3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81 not found: ID does not exist" containerID="3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81"
Oct 30 19:39:04.603479 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.603433    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81"} err="failed to get container status \"3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81\": rpc error: code = NotFound desc = could not find container \"3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81\": container with ID starting with 3aabec57c4871f8b11e8a48df7341606f7bc11eea2eb58c35654b7429add1f81 not found: ID does not exist"
Oct 30 19:39:04.746863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.746834    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 19:39:04.798425 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:04.795018    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Oct 30 19:39:06.055985 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:06.055949    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7241f51b-3078-4f88-b21f-8bc14c14afb1" path="/var/lib/kubelet/pods/7241f51b-3078-4f88-b21f-8bc14c14afb1/volumes"
Oct 30 19:39:13.868807 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:13.868723    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:39:16.565466 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.565412    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:39:16.593380 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.593353    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e23a5c6-c8e7-490a-a7df-316328e4096e-kube-api-access-fqvx4\" (UniqueName: \"kubernetes.io/projected/5e23a5c6-c8e7-490a-a7df-316328e4096e-kube-api-access-fqvx4\") pod \"5e23a5c6-c8e7-490a-a7df-316328e4096e\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") "
Oct 30 19:39:16.593480 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.593392    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e23a5c6-c8e7-490a-a7df-316328e4096e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e23a5c6-c8e7-490a-a7df-316328e4096e-tls-certs\") pod \"5e23a5c6-c8e7-490a-a7df-316328e4096e\" (UID: \"5e23a5c6-c8e7-490a-a7df-316328e4096e\") "
Oct 30 19:39:16.595517 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.595481    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e23a5c6-c8e7-490a-a7df-316328e4096e-tls-certs" pod "5e23a5c6-c8e7-490a-a7df-316328e4096e" (UID: "5e23a5c6-c8e7-490a-a7df-316328e4096e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:39:16.595611 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.595556    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e23a5c6-c8e7-490a-a7df-316328e4096e-kube-api-access-fqvx4" pod "5e23a5c6-c8e7-490a-a7df-316328e4096e" (UID: "5e23a5c6-c8e7-490a-a7df-316328e4096e"). InnerVolumeSpecName "kube-api-access-fqvx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:16.642665 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.642637    5025 generic.go:356] "Generic (PLEG): container finished" podID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerID="fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32" exitCode=0
Oct 30 19:39:16.642771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.642687    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"
Oct 30 19:39:16.642771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.642714    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" event={"ID":"5e23a5c6-c8e7-490a-a7df-316328e4096e","Type":"ContainerDied","Data":"fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32"}
Oct 30 19:39:16.642771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.642770    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9" event={"ID":"5e23a5c6-c8e7-490a-a7df-316328e4096e","Type":"ContainerDied","Data":"490bb91c1c4f928e79bb9f881501a9081689ee438ab451d1fef60f0915c949f8"}
Oct 30 19:39:16.642887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.642792    5025 scope.go:122] "RemoveContainer" containerID="fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32"
Oct 30 19:39:16.651709 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.651691    5025 scope.go:122] "RemoveContainer" containerID="fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32"
Oct 30 19:39:16.651957 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:39:16.651936    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32\": container with ID starting with fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32 not found: ID does not exist" containerID="fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32"
Oct 30 19:39:16.652010 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.651964    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32"} err="failed to get container status \"fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32\": rpc error: code = NotFound desc = could not find container \"fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32\": container with ID starting with fda25cc12d70a3bc016ae049f24ab704e0f0887fbe04695edc35c0cf87af6b32 not found: ID does not exist"
Oct 30 19:39:16.694741 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.694719    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqvx4\" (UniqueName: \"kubernetes.io/projected/5e23a5c6-c8e7-490a-a7df-316328e4096e-kube-api-access-fqvx4\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:16.694741 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.694742    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e23a5c6-c8e7-490a-a7df-316328e4096e-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:16.784732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.784705    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"]
Oct 30 19:39:16.789994 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:16.789970    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheglpj9"]
Oct 30 19:39:18.055818 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:18.055784    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" path="/var/lib/kubelet/pods/5e23a5c6-c8e7-490a-a7df-316328e4096e/volumes"
Oct 30 19:39:28.081858 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:39:28.081818    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:39:33.372740 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.372703    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"]
Oct 30 19:39:33.373243 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373226    5025 cpu_manager.go:414] "containerMap: removing container" podUID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerName="storage-initializer"
Oct 30 19:39:33.373286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373245    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerName="storage-initializer"
Oct 30 19:39:33.373286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373271    5025 cpu_manager.go:414] "containerMap: removing container" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main"
Oct 30 19:39:33.373286 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373279    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main"
Oct 30 19:39:33.373410 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373310    5025 cpu_manager.go:414] "containerMap: removing container" podUID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerName="main"
Oct 30 19:39:33.373410 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373336    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerName="main"
Oct 30 19:39:33.373478 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373435    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241f51b-3078-4f88-b21f-8bc14c14afb1" containerName="main"
Oct 30 19:39:33.373478 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.373452    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e23a5c6-c8e7-490a-a7df-316328e4096e" containerName="main"
Oct 30 19:39:33.376860 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.376835    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.417335 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.417297    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"]
Oct 30 19:39:33.435522 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435492    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-model-cache\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.435660 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435526    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90c184c-0553-4d95-a020-8d499b159879-tls-certs\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.435660 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435554    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-tmp-dir\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.435776 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435642    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hq4p\" (UniqueName: \"kubernetes.io/projected/e90c184c-0553-4d95-a020-8d499b159879-kube-api-access-2hq4p\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.435814 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435778    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90c184c-0553-4d95-a020-8d499b159879-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.435814 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435806    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-home\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.435892 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435878    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-dshm\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.435930 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.435906    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.463078 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.463051    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"kube-root-ca.crt\""
Oct 30 19:39:33.463223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.463137    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"seaweedfs-s3-creds\""
Oct 30 19:39:33.463223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.463141    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:39:33.463223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.463170    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"openshift-service-ca.crt\""
Oct 30 19:39:33.463223 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.463179    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"default-dockercfg-72njd\""
Oct 30 19:39:33.463460 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.463228    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-kserve-self-signed-certs\""
Oct 30 19:39:33.536898 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.536860    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.536914    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-model-cache\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.536944    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90c184c-0553-4d95-a020-8d499b159879-tls-certs\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.536986    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-tmp-dir\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.537016    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hq4p\" (UniqueName: \"kubernetes.io/projected/e90c184c-0553-4d95-a020-8d499b159879-kube-api-access-2hq4p\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.537043    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90c184c-0553-4d95-a020-8d499b159879-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.537072    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-home\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.537144    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-dshm\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.537358    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-kserve-provision-location\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537581 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.537530    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-model-cache\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.537774 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.537746    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-home\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.538028 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.538007    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-tmp-dir\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.538100 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.538036    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90c184c-0553-4d95-a020-8d499b159879-cabundle-cert\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.539619 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.539597    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-dshm\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.539732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.539670    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90c184c-0553-4d95-a020-8d499b159879-tls-certs\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.600676 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.600649    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hq4p\" (UniqueName: \"kubernetes.io/projected/e90c184c-0553-4d95-a020-8d499b159879-kube-api-access-2hq4p\") pod \"tls-verification-test-kserve-prefill-769794f89-qc44x\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.691983 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.691908    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:33.800130 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.800099    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"]
Oct 30 19:39:33.803918 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.803893    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:33.839678 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.839648    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfll\" (UniqueName: \"kubernetes.io/projected/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-kube-api-access-hkfll\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:33.839807 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.839696    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:33.856247 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.856201    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-e4cad0c3\"/\"tls-verification-test-epp-sa-dockercfg-7rv8r\""
Oct 30 19:39:33.857743 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.857718    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"]
Oct 30 19:39:33.941188 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.941152    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfll\" (UniqueName: \"kubernetes.io/projected/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-kube-api-access-hkfll\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:33.941404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.941192    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:33.943854 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.943793    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:33.997731 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:33.997697    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfll\" (UniqueName: \"kubernetes.io/projected/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-kube-api-access-hkfll\") pod \"tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") " pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:34.116941 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.116900    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:34.171012 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.170985    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"]
Oct 30 19:39:34.173748 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:39:34.173713    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90c184c_0553_4d95_a020_8d499b159879.slice/crio-7b8976e2e99bf76e95244028a467306fa3280f871a3f734703739d96655d73d6 WatchSource:0}: Error finding container 7b8976e2e99bf76e95244028a467306fa3280f871a3f734703739d96655d73d6: Status 404 returned error can't find the container with id 7b8976e2e99bf76e95244028a467306fa3280f871a3f734703739d96655d73d6
Oct 30 19:39:34.485222 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.485139    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"]
Oct 30 19:39:34.490880 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:39:34.490837    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81858cf0_3751_4e2d_8dfc_1e93ec4b1927.slice/crio-42c7dca4483a831337b4ac7eb94e91311d550b5bb363c3cc28c342ac0a085982 WatchSource:0}: Error finding container 42c7dca4483a831337b4ac7eb94e91311d550b5bb363c3cc28c342ac0a085982: Status 404 returned error can't find the container with id 42c7dca4483a831337b4ac7eb94e91311d550b5bb363c3cc28c342ac0a085982
Oct 30 19:39:34.733401 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.733349    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" event={"ID":"81858cf0-3751-4e2d-8dfc-1e93ec4b1927","Type":"ContainerStarted","Data":"7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2"}
Oct 30 19:39:34.733607 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.733494    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" event={"ID":"81858cf0-3751-4e2d-8dfc-1e93ec4b1927","Type":"ContainerStarted","Data":"42c7dca4483a831337b4ac7eb94e91311d550b5bb363c3cc28c342ac0a085982"}
Oct 30 19:39:34.733607 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.733542    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:39:34.735149 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.735117    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" event={"ID":"e90c184c-0553-4d95-a020-8d499b159879","Type":"ContainerStarted","Data":"8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711"}
Oct 30 19:39:34.735297 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.735172    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" event={"ID":"e90c184c-0553-4d95-a020-8d499b159879","Type":"ContainerStarted","Data":"7b8976e2e99bf76e95244028a467306fa3280f871a3f734703739d96655d73d6"}
Oct 30 19:39:34.845127 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:34.845075    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" podStartSLOduration=1.845063758 podStartE2EDuration="1.845063758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:34.844369084 +0000 UTC m=+3756.924955291" watchObservedRunningTime="2026-10-30 19:39:34.845063758 +0000 UTC m=+3756.925649965"
Oct 30 19:39:35.972050 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:35.972012    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"]
Oct 30 19:39:35.972628 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:35.972230    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy" containerID="cri-o://ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83" gracePeriod=30
Oct 30 19:39:36.747923 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:36.747884    5025 generic.go:356] "Generic (PLEG): container finished" podID="e90c184c-0553-4d95-a020-8d499b159879" containerID="8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711" exitCode=0
Oct 30 19:39:36.748091 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:36.747929    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" event={"ID":"e90c184c-0553-4d95-a020-8d499b159879","Type":"ContainerDied","Data":"8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711"}
Oct 30 19:39:37.755623 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:37.755583    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" event={"ID":"e90c184c-0553-4d95-a020-8d499b159879","Type":"ContainerStarted","Data":"c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b"}
Oct 30 19:39:37.965368 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:37.965292    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podStartSLOduration=2.684463598 podStartE2EDuration="4.965277011s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.280813413s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:37.964604947 +0000 UTC m=+3760.045191153" watchObservedRunningTime="2026-10-30 19:39:37.965277011 +0000 UTC m=+3760.045863217"
Oct 30 19:39:41.234741 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.234715    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:39:41.320577 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320490    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-podinfo\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320577 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320524    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-certs\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320577 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320544    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-token\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320577 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320578    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-socket\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320613    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-envoy\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320647    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istiod-ca-cert\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320702    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-kube-api-access-k6tg2\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-kube-api-access-k6tg2\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320754    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-credential-socket\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320815    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-data\") pod \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\" (UID: \"f7abe904-2a3d-4bad-8b33-e2f4b62af417\") "
Oct 30 19:39:41.320922 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.320830    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-certs" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.321209 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.321170    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-socket" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.321209 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.321195    5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.321351 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.321228    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-credential-socket" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.321430 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.321406    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-data" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.321612 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.321480    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istiod-ca-cert" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:39:41.323118 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.323088    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-kube-api-access-k6tg2" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "kube-api-access-k6tg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:41.323118 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.323099    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-token" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:39:41.323285 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.323262    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-podinfo" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 19:39:41.323370 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.323304    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-envoy" pod "f7abe904-2a3d-4bad-8b33-e2f4b62af417" (UID: "f7abe904-2a3d-4bad-8b33-e2f4b62af417"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:39:41.422511 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422481    5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-podinfo\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.422511 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422507    5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-token\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.422511 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422517    5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-workload-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.422742 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422526    5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-envoy\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.422742 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422537    5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istiod-ca-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.422742 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422546    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6tg2\" (UniqueName: \"kubernetes.io/projected/f7abe904-2a3d-4bad-8b33-e2f4b62af417-kube-api-access-k6tg2\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.422742 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422556    5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-credential-socket\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.422742 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.422564    5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7abe904-2a3d-4bad-8b33-e2f4b62af417-istio-data\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:39:41.777983 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.777947    5025 generic.go:356] "Generic (PLEG): container finished" podID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerID="ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83" exitCode=0
Oct 30 19:39:41.778163 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.778005    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" event={"ID":"f7abe904-2a3d-4bad-8b33-e2f4b62af417","Type":"ContainerDied","Data":"ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83"}
Oct 30 19:39:41.778163 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.778074    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs" event={"ID":"f7abe904-2a3d-4bad-8b33-e2f4b62af417","Type":"ContainerDied","Data":"e7cf53a4cd19c15a7ea28aac473c810e4f1400e8d754077a2b7479e37b6a8989"}
Oct 30 19:39:41.778163 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.778008    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"
Oct 30 19:39:41.778163 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.778097    5025 scope.go:122] "RemoveContainer" containerID="ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83"
Oct 30 19:39:41.787555 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.787535    5025 scope.go:122] "RemoveContainer" containerID="a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064"
Oct 30 19:39:41.796471 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.796451    5025 scope.go:122] "RemoveContainer" containerID="ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83"
Oct 30 19:39:41.796739 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:39:41.796720    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83\": container with ID starting with ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83 not found: ID does not exist" containerID="ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83"
Oct 30 19:39:41.796816 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.796753    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83"} err="failed to get container status \"ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83\": rpc error: code = NotFound desc = could not find container \"ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83\": container with ID starting with ff9b03e23d670cc7c48f56982d33b5cdc49892a0563b7ebd98f8794fb495fe83 not found: ID does not exist"
Oct 30 19:39:41.796816 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.796777    5025 scope.go:122] "RemoveContainer" containerID="a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064"
Oct 30 19:39:41.797053 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:39:41.797030    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064\": container with ID starting with a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064 not found: ID does not exist" containerID="a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064"
Oct 30 19:39:41.797092 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.797066    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064"} err="failed to get container status \"a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064\": rpc error: code = NotFound desc = could not find container \"a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064\": container with ID starting with a758275208af5e209d8bfa015fce0250084401a0de26d1c5a8634413d2351064 not found: ID does not exist"
Oct 30 19:39:41.937568 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.937529    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"]
Oct 30 19:39:41.984956 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:41.984917    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-based-routing-only-fae765e6/model-routing-only-gateway-openshift-default-7d46dbd96b-8rrzs"]
Oct 30 19:39:42.057230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:42.057151    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" path="/var/lib/kubelet/pods/f7abe904-2a3d-4bad-8b33-e2f4b62af417/volumes"
Oct 30 19:39:43.692617 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:43.692569    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:43.693125 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:43.692655    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:39:43.693980 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:43.693948    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:39:45.369970 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.369936    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"]
Oct 30 19:39:45.370465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.370341    5025 cpu_manager.go:414] "containerMap: removing container" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy"
Oct 30 19:39:45.370465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.370354    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy"
Oct 30 19:39:45.370465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.370370    5025 cpu_manager.go:414] "containerMap: removing container" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy"
Oct 30 19:39:45.370465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.370376    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy"
Oct 30 19:39:45.370465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.370444    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy"
Oct 30 19:39:45.370465 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.370453    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7abe904-2a3d-4bad-8b33-e2f4b62af417" containerName="istio-proxy"
Oct 30 19:39:45.376470 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.376448    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.422050 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.422018    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"]
Oct 30 19:39:45.426942 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.426914    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Oct 30 19:39:45.459896 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.459858    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4076d08b-adae-4c20-87f9-a20f0ff6360b-tls-certs\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.460012 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.459902    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdvfh\" (UniqueName: \"kubernetes.io/projected/4076d08b-adae-4c20-87f9-a20f0ff6360b-kube-api-access-fdvfh\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.460012 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.459997    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-tmp-dir\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.460107 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.460044    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-dshm\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.460152 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.460110    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4076d08b-adae-4c20-87f9-a20f0ff6360b-cabundle-cert\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.460190 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.460168    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-home\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.460234 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.460190    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-kserve-provision-location\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.460267 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.460233    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-model-cache\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.467644 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.467620    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Oct 30 19:39:45.467766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.467669    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-m94j9\""
Oct 30 19:39:45.467766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.467733    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 19:39:45.467766 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.467749    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Oct 30 19:39:45.467914 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.467753    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Oct 30 19:39:45.560751 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.560719    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-dshm\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.560884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.560766    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4076d08b-adae-4c20-87f9-a20f0ff6360b-cabundle-cert\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.560884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.560816    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-home\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.560884 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.560859    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-kserve-provision-location\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561006 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.560895    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-model-cache\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561006 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.560965    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4076d08b-adae-4c20-87f9-a20f0ff6360b-tls-certs\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561006 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.560994    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdvfh\" (UniqueName: \"kubernetes.io/projected/4076d08b-adae-4c20-87f9-a20f0ff6360b-kube-api-access-fdvfh\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561152 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.561043    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-tmp-dir\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561313 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.561288    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-home\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561423 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.561313    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-kserve-provision-location\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561423 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.561391    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-model-cache\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561543 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.561461    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-tmp-dir\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.561618 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.561592    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4076d08b-adae-4c20-87f9-a20f0ff6360b-cabundle-cert\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.563197 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.563165    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-dshm\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.563743 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.563715    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4076d08b-adae-4c20-87f9-a20f0ff6360b-tls-certs\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.619333 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.619298    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdvfh\" (UniqueName: \"kubernetes.io/projected/4076d08b-adae-4c20-87f9-a20f0ff6360b-kube-api-access-fdvfh\") pod \"prestop-hook-test-kserve-698749d5d6-clbfp\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:45.695272 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:45.695208    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:46.141093 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:46.141054    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"]
Oct 30 19:39:46.146743 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:39:46.146714    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4076d08b_adae_4c20_87f9_a20f0ff6360b.slice/crio-f4f27137bbb88b99110bc74fe9e8939c838b0359fe37771999e8e1f274de2ecc WatchSource:0}: Error finding container f4f27137bbb88b99110bc74fe9e8939c838b0359fe37771999e8e1f274de2ecc: Status 404 returned error can't find the container with id f4f27137bbb88b99110bc74fe9e8939c838b0359fe37771999e8e1f274de2ecc
Oct 30 19:39:46.807658 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:46.807613    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" event={"ID":"4076d08b-adae-4c20-87f9-a20f0ff6360b","Type":"ContainerStarted","Data":"b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac"}
Oct 30 19:39:46.808175 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:46.807678    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" event={"ID":"4076d08b-adae-4c20-87f9-a20f0ff6360b","Type":"ContainerStarted","Data":"f4f27137bbb88b99110bc74fe9e8939c838b0359fe37771999e8e1f274de2ecc"}
Oct 30 19:39:48.828765 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:48.828685    5025 generic.go:356] "Generic (PLEG): container finished" podID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerID="b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac" exitCode=0
Oct 30 19:39:48.829152 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:48.828761    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" event={"ID":"4076d08b-adae-4c20-87f9-a20f0ff6360b","Type":"ContainerDied","Data":"b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac"}
Oct 30 19:39:49.835266 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:49.835230    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" event={"ID":"4076d08b-adae-4c20-87f9-a20f0ff6360b","Type":"ContainerStarted","Data":"b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd"}
Oct 30 19:39:50.046552 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:50.046501    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podStartSLOduration=2.706055 podStartE2EDuration="5.046485384s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.340430384s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:49.997682771 +0000 UTC m=+3772.078268996" watchObservedRunningTime="2026-10-30 19:39:50.046485384 +0000 UTC m=+3772.127071568"
Oct 30 19:39:50.695455 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:50.695422    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:53.692355 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:53.692295    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:39:55.278567 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.278530    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"]
Oct 30 19:39:55.283804 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.283778    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.326857 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.326824    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"]
Oct 30 19:39:55.332136 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.332113    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.344991 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.344961    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 19:39:55.345112 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.345009    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-dstgc\""
Oct 30 19:39:55.378211 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.378176    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"]
Oct 30 19:39:55.429642 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.429609    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"]
Oct 30 19:39:55.457217 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457186    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.457417 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457241    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457417 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457337    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.457417 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457395    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.457554 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457418    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcqp\" (UniqueName: \"kubernetes.io/projected/b2e16b4b-18e8-45aa-a967-185abc5a52ca-kube-api-access-cjcqp\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.457554 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457455    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457636 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457534    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f5514c-72f3-4027-b661-8cad46b22185-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457636 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457579    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f5514c-72f3-4027-b661-8cad46b22185-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457636 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457603    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457638    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.457744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457692    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457710    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.457744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457728    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f5514c-72f3-4027-b661-8cad46b22185-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457898 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457744    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457898 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457798    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.457973 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457898    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6d8w\" (UniqueName: \"kubernetes.io/projected/25f5514c-72f3-4027-b661-8cad46b22185-kube-api-access-w6d8w\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.457973 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457941    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.458230 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.457971    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.559346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559242    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.559346 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559300    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559348    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559388    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559419    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559474    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559498    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcqp\" (UniqueName: \"kubernetes.io/projected/b2e16b4b-18e8-45aa-a967-185abc5a52ca-kube-api-access-cjcqp\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559533    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559560    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f5514c-72f3-4027-b661-8cad46b22185-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.559600 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559587    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f5514c-72f3-4027-b661-8cad46b22185-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559610    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559631    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559671    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559694    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559720    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f5514c-72f3-4027-b661-8cad46b22185-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559752    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559770    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.560041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.559793    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.560480 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.560053    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560480 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.560095    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560728 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.560703    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560862 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.560833    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25f5514c-72f3-4027-b661-8cad46b22185-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.560938 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.560739    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.561001 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.560975    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.561111 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.561090    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.561254 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.561232    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.561434 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.561415    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6d8w\" (UniqueName: \"kubernetes.io/projected/25f5514c-72f3-4027-b661-8cad46b22185-kube-api-access-w6d8w\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.561689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.561658    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.562618 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.562592    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.562734 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.562683    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.562975 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.562953    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25f5514c-72f3-4027-b661-8cad46b22185-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.563632 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.563614    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25f5514c-72f3-4027-b661-8cad46b22185-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.618057 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.618019    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcqp\" (UniqueName: \"kubernetes.io/projected/b2e16b4b-18e8-45aa-a967-185abc5a52ca-kube-api-access-cjcqp\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.618309 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.618022    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e16b4b-18e8-45aa-a967-185abc5a52ca-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-gtvkd\" (UID: \"b2e16b4b-18e8-45aa-a967-185abc5a52ca\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:55.618309 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.618245    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6d8w\" (UniqueName: \"kubernetes.io/projected/25f5514c-72f3-4027-b661-8cad46b22185-kube-api-access-w6d8w\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.618833 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.618813    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25f5514c-72f3-4027-b661-8cad46b22185-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-ztv2x\" (UID: \"25f5514c-72f3-4027-b661-8cad46b22185\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.646952 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.646925    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:55.695753 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.695708    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:39:55.698076 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.697745    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:39:55.898069 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:55.898035    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:56.076338 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.076295    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"]
Oct 30 19:39:56.082577 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:39:56.082546    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25f5514c_72f3_4027_b661_8cad46b22185.slice/crio-592a85fa8a05c01d0872f6cfc573650e5b97a30d4a462fac623c5504c6c8e243 WatchSource:0}: Error finding container 592a85fa8a05c01d0872f6cfc573650e5b97a30d4a462fac623c5504c6c8e243: Status 404 returned error can't find the container with id 592a85fa8a05c01d0872f6cfc573650e5b97a30d4a462fac623c5504c6c8e243
Oct 30 19:39:56.084986 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.084941    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:56.085062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.085039    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:56.085106 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.085086    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:56.285530 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.285492    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"]
Oct 30 19:39:56.293356 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:39:56.293308    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e16b4b_18e8_45aa_a967_185abc5a52ca.slice/crio-05f9b6440f345d2524698841d0dd7cf495c333470da5b0fa59944cc9b8f09f7b WatchSource:0}: Error finding container 05f9b6440f345d2524698841d0dd7cf495c333470da5b0fa59944cc9b8f09f7b: Status 404 returned error can't find the container with id 05f9b6440f345d2524698841d0dd7cf495c333470da5b0fa59944cc9b8f09f7b
Oct 30 19:39:56.296264 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.296230    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:56.296362 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.296314    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:56.296418 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.296374    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:39:56.919349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.903055    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" event={"ID":"b2e16b4b-18e8-45aa-a967-185abc5a52ca","Type":"ContainerStarted","Data":"e95dce0b124d8be3be377feaaa711ef24e0b5824c9a2466adea849712dfded8c"}
Oct 30 19:39:56.919349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.903126    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" event={"ID":"b2e16b4b-18e8-45aa-a967-185abc5a52ca","Type":"ContainerStarted","Data":"05f9b6440f345d2524698841d0dd7cf495c333470da5b0fa59944cc9b8f09f7b"}
Oct 30 19:39:56.919349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.906076    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" event={"ID":"25f5514c-72f3-4027-b661-8cad46b22185","Type":"ContainerStarted","Data":"bb51476d3280350c249e8f50b70e2c9dd84c213893b0f89091742a30ea5f4d73"}
Oct 30 19:39:56.919349 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:56.906134    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" event={"ID":"25f5514c-72f3-4027-b661-8cad46b22185","Type":"ContainerStarted","Data":"592a85fa8a05c01d0872f6cfc573650e5b97a30d4a462fac623c5504c6c8e243"}
Oct 30 19:39:57.141035 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.140978    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" podStartSLOduration=2.140961932 podStartE2EDuration="2.140961932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:57.140118673 +0000 UTC m=+3779.220704881" watchObservedRunningTime="2026-10-30 19:39:57.140961932 +0000 UTC m=+3779.221548139"
Oct 30 19:39:57.142027 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.141990    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" podStartSLOduration=2.141982708 podStartE2EDuration="2.141982708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:57.02666038 +0000 UTC m=+3779.107246590" watchObservedRunningTime="2026-10-30 19:39:57.141982708 +0000 UTC m=+3779.222568915"
Oct 30 19:39:57.647514 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.647470    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:39:57.649392 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.649363    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:39:57.649532 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.649412    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" podUID="25f5514c-72f3-4027-b661-8cad46b22185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:39:57.904184 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.899714    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:39:57.904387 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.904276    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:39:57.904387 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:57.904350    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" podUID="b2e16b4b-18e8-45aa-a967-185abc5a52ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:39:58.186524 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:39:58.180729    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:39:58.647860 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:58.647810    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:39:58.648419 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:58.647882    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" podUID="25f5514c-72f3-4027-b661-8cad46b22185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:39:58.904687 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:58.904124    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:39:58.904687 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:58.904185    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" podUID="b2e16b4b-18e8-45aa-a967-185abc5a52ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:39:59.647365 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:59.647312    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:39:59.647601 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:59.647387    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" podUID="25f5514c-72f3-4027-b661-8cad46b22185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:39:59.910400 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:59.901748    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:39:59.910400 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:39:59.901813    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" podUID="b2e16b4b-18e8-45aa-a967-185abc5a52ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:40:00.647871 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:00.647830    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:40:00.648099 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:00.647886    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" podUID="25f5514c-72f3-4027-b661-8cad46b22185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:40:00.899406 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:00.898830    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:40:00.899406 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:00.898907    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" podUID="b2e16b4b-18e8-45aa-a967-185abc5a52ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:40:01.647425 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:01.647388    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 19:40:01.647968 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:01.647443    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x" podUID="25f5514c-72f3-4027-b661-8cad46b22185" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 19:40:01.903816 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:01.903712    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 19:40:01.904070 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:01.904040    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd" podUID="b2e16b4b-18e8-45aa-a967-185abc5a52ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 19:40:02.653198 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:02.653161    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:40:02.653719 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:02.653623    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:40:02.654503 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:02.654471    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-ztv2x"
Oct 30 19:40:02.902809 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:02.902783    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:40:02.903105 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:02.903079    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:40:02.903774 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:02.903733    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-gtvkd"
Oct 30 19:40:03.692758 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:03.692707    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:40:05.695813 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:05.695767    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:40:05.742233 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:05.742200    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:40:10.394500 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.394463    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"]
Oct 30 19:40:10.398475 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.398453    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457032 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.456997    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt4t2\" (UniqueName: \"kubernetes.io/projected/137b793a-25e8-4205-bedb-498b6b1246d1-kube-api-access-bt4t2\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457197 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457047    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b793a-25e8-4205-bedb-498b6b1246d1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457197 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457067    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137b793a-25e8-4205-bedb-498b6b1246d1-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457197 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457174    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457197 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457196    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457392 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457220    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457392 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457312    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457392 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457375    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b793a-25e8-4205-bedb-498b6b1246d1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.457506 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.457406    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.498800 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.498763    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"]
Oct 30 19:40:10.558378 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558310    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558407    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558462    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558500    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b793a-25e8-4205-bedb-498b6b1246d1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558566 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558550    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558585    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt4t2\" (UniqueName: \"kubernetes.io/projected/137b793a-25e8-4205-bedb-498b6b1246d1-kube-api-access-bt4t2\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558644    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b793a-25e8-4205-bedb-498b6b1246d1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558679    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137b793a-25e8-4205-bedb-498b6b1246d1-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.558801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558739    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.559034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558798    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-workload-certs\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.559034 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.558905    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-istio-data\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.559171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.559066    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-workload-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.559171 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.559144    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-credential-socket\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.559578 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.559556    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/137b793a-25e8-4205-bedb-498b6b1246d1-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.560944 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.560923    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/137b793a-25e8-4205-bedb-498b6b1246d1-istio-envoy\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.561680 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.561650    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/137b793a-25e8-4205-bedb-498b6b1246d1-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.616102 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.616063    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/137b793a-25e8-4205-bedb-498b6b1246d1-istio-token\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.616289 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.616074    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt4t2\" (UniqueName: \"kubernetes.io/projected/137b793a-25e8-4205-bedb-498b6b1246d1-kube-api-access-bt4t2\") pod \"openshift-ai-inference-openshift-default-5469fc87dd-2sqns\" (UID: \"137b793a-25e8-4205-bedb-498b6b1246d1\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:10.711621 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:10.711525    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:11.145310 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:11.145276    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"]
Oct 30 19:40:11.184216 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:40:11.184178    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod137b793a_25e8_4205_bedb_498b6b1246d1.slice/crio-23eb62e6eb9ffce1dceaa7d08d7c51c1fa3f24a1ead28249aee3bb549b8e80c3 WatchSource:0}: Error finding container 23eb62e6eb9ffce1dceaa7d08d7c51c1fa3f24a1ead28249aee3bb549b8e80c3: Status 404 returned error can't find the container with id 23eb62e6eb9ffce1dceaa7d08d7c51c1fa3f24a1ead28249aee3bb549b8e80c3
Oct 30 19:40:11.186667 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:11.186635    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:40:11.186753 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:11.186721    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:40:11.186794 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:11.186750    5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 19:40:12.011645 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:12.011608    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" event={"ID":"137b793a-25e8-4205-bedb-498b6b1246d1","Type":"ContainerStarted","Data":"9acdc8b45510425560182d0dd3b7f9881648e231d2ed53b5d870e9b221029a56"}
Oct 30 19:40:12.012164 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:12.011660    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" event={"ID":"137b793a-25e8-4205-bedb-498b6b1246d1","Type":"ContainerStarted","Data":"23eb62e6eb9ffce1dceaa7d08d7c51c1fa3f24a1ead28249aee3bb549b8e80c3"}
Oct 30 19:40:12.147709 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:12.147555    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" podStartSLOduration=2.147541192 podStartE2EDuration="2.147541192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:12.145856335 +0000 UTC m=+3794.226442544" watchObservedRunningTime="2026-10-30 19:40:12.147541192 +0000 UTC m=+3794.228127398"
Oct 30 19:40:12.712043 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:12.712009    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:12.713770 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:12.713745    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2sqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:40:12.713924 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:12.713788    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" podUID="137b793a-25e8-4205-bedb-498b6b1246d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:40:13.693041 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:13.692989    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:40:13.712047 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:13.712012    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2sqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:40:13.712187 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:13.712075    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" podUID="137b793a-25e8-4205-bedb-498b6b1246d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:40:14.712299 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:14.712256    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2sqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:40:14.712713 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:14.712347    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" podUID="137b793a-25e8-4205-bedb-498b6b1246d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:40:15.696046 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:15.695995    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:40:15.712624 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:15.712564    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2sqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:40:15.712996 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:15.712632    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" podUID="137b793a-25e8-4205-bedb-498b6b1246d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:40:16.717393 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:16.716894    5025 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-default-5469fc87dd-2sqns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 19:40:16.717393 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:16.716985    5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns" podUID="137b793a-25e8-4205-bedb-498b6b1246d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 19:40:17.716639 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:17.716602    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:17.717002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:17.716971    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:17.717842 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:17.717823    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-5469fc87dd-2sqns"
Oct 30 19:40:23.692792 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:23.692737    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:40:25.696057 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:25.696007    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:40:28.084224 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:40:28.084183    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:40:33.693384 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:33.693317    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:40:35.696023 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:35.695975    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:40:43.692698 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:43.692595    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:40:45.695744 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:45.695688    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:40:53.692293 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:53.692242    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:40:55.696491 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:40:55.696444    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:40:58.091303 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:40:58.091257    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:41:03.692360 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:03.692306    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:41:05.696145 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:05.696101    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:41:13.692771 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:13.692720    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:41:15.696133 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:15.696089    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:41:23.693296 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:23.693255    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:41:25.695863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:25.695821    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:41:28.083438 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:41:28.083398    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:41:33.692481 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:33.692428    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:41:35.695661 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:35.695617    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:41:43.692485 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:43.692446    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:41:45.695779 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:45.695739    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:41:53.693176 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:53.693127    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 30 19:41:55.696635 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:55.696591    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:41:58.082379 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:41:58.082337    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:41:58.557586 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:58.557555    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:41:58.565224 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:41:58.565198    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:42:03.708441 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:03.708406    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:42:03.720073 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:03.720047    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:42:05.696407 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:05.696358    5025 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:42:15.706115 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:15.706036    5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:42:15.714015 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:15.713988    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:42:19.184979 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:19.184943    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"]
Oct 30 19:42:28.081789 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:42:28.081751    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:42:34.207679 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.207641    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" containerID="cri-o://b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd" gracePeriod=45
Oct 30 19:42:34.707210 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.707165    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Oct 30 19:42:34.952062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.952035    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:42:34.996786 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996709    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-home\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-home\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.996786 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996751    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4076d08b-adae-4c20-87f9-a20f0ff6360b-kube-api-access-fdvfh\" (UniqueName: \"kubernetes.io/projected/4076d08b-adae-4c20-87f9-a20f0ff6360b-kube-api-access-fdvfh\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.997002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996798    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-kserve-provision-location\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.997002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996822    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-dshm\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.997002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996844    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-model-cache\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.997002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996888    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-tmp-dir\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.997002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996941    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4076d08b-adae-4c20-87f9-a20f0ff6360b-tls-certs\" (UniqueName: \"kubernetes.io/secret/4076d08b-adae-4c20-87f9-a20f0ff6360b-tls-certs\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.997002 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.996968    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4076d08b-adae-4c20-87f9-a20f0ff6360b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4076d08b-adae-4c20-87f9-a20f0ff6360b-cabundle-cert\") pod \"4076d08b-adae-4c20-87f9-a20f0ff6360b\" (UID: \"4076d08b-adae-4c20-87f9-a20f0ff6360b\") "
Oct 30 19:42:34.997351 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.997161    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-model-cache" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:34.997534 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.997494    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4076d08b-adae-4c20-87f9-a20f0ff6360b-cabundle-cert" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:42:34.997656 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.997554    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-home" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:34.999416 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.999299    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4076d08b-adae-4c20-87f9-a20f0ff6360b-kube-api-access-fdvfh" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "kube-api-access-fdvfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:42:34.999535 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.999497    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-dshm" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:34.999793 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:34.999773    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4076d08b-adae-4c20-87f9-a20f0ff6360b-tls-certs" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:42:35.009072 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.009047    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-tmp-dir" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:35.021728 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.021704    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-kserve-provision-location" pod "4076d08b-adae-4c20-87f9-a20f0ff6360b" (UID: "4076d08b-adae-4c20-87f9-a20f0ff6360b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:42:35.098492 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098466    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4076d08b-adae-4c20-87f9-a20f0ff6360b-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.098492 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098487    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4076d08b-adae-4c20-87f9-a20f0ff6360b-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.098641 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098498    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.098641 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098507    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdvfh\" (UniqueName: \"kubernetes.io/projected/4076d08b-adae-4c20-87f9-a20f0ff6360b-kube-api-access-fdvfh\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.098641 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098517    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.098641 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098529    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.098641 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098542    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.098641 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.098552    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4076d08b-adae-4c20-87f9-a20f0ff6360b-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:42:35.780592 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.780561    5025 generic.go:356] "Generic (PLEG): container finished" podID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerID="b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd" exitCode=0
Oct 30 19:42:35.781071 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.780612    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"
Oct 30 19:42:35.781071 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.780650    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" event={"ID":"4076d08b-adae-4c20-87f9-a20f0ff6360b","Type":"ContainerDied","Data":"b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd"}
Oct 30 19:42:35.781071 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.780718    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp" event={"ID":"4076d08b-adae-4c20-87f9-a20f0ff6360b","Type":"ContainerDied","Data":"f4f27137bbb88b99110bc74fe9e8939c838b0359fe37771999e8e1f274de2ecc"}
Oct 30 19:42:35.781071 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.780754    5025 scope.go:122] "RemoveContainer" containerID="b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd"
Oct 30 19:42:35.790825 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.790802    5025 scope.go:122] "RemoveContainer" containerID="b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac"
Oct 30 19:42:35.798692 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.798674    5025 scope.go:122] "RemoveContainer" containerID="b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd"
Oct 30 19:42:35.798926 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:42:35.798911    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd\": container with ID starting with b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd not found: ID does not exist" containerID="b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd"
Oct 30 19:42:35.798975 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.798942    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd"} err="failed to get container status \"b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd\": rpc error: code = NotFound desc = could not find container \"b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd\": container with ID starting with b3e430214fb858e7a764a78302e4105b9bdc6c3e2744b93e4fff7b1f7b526ecd not found: ID does not exist"
Oct 30 19:42:35.798975 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.798959    5025 scope.go:122] "RemoveContainer" containerID="b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac"
Oct 30 19:42:35.799182 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:42:35.799164    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac\": container with ID starting with b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac not found: ID does not exist" containerID="b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac"
Oct 30 19:42:35.799235 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.799191    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac"} err="failed to get container status \"b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac\": rpc error: code = NotFound desc = could not find container \"b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac\": container with ID starting with b10135ea4195604e47df9d2d560070faf7158283cb696c54ba9f1580b56073ac not found: ID does not exist"
Oct 30 19:42:35.941426 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.941396    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"]
Oct 30 19:42:35.988698 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:35.988668    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-698749d5d6-clbfp"]
Oct 30 19:42:36.056459 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:42:36.056385    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" path="/var/lib/kubelet/pods/4076d08b-adae-4c20-87f9-a20f0ff6360b/volumes"
Oct 30 19:42:58.081450 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:42:58.081414    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:43:17.662449 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:17.662412    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"]
Oct 30 19:43:17.698254 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:17.698224    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"]
Oct 30 19:43:25.742122 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:25.742081    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:28.081312 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:43:28.081272    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:43:32.682716 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:32.682645    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main" containerID="cri-o://7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2" gracePeriod=45
Oct 30 19:43:32.720839 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:32.720803    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main" containerID="cri-o://c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b" gracePeriod=45
Oct 30 19:43:33.380544 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.380521    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:43:33.439404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439315    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90c184c-0553-4d95-a020-8d499b159879-tls-certs\" (UniqueName: \"kubernetes.io/secret/e90c184c-0553-4d95-a020-8d499b159879-tls-certs\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439386    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-model-cache\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439585 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439439    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-dshm\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439585 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439474    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90c184c-0553-4d95-a020-8d499b159879-kube-api-access-2hq4p\" (UniqueName: \"kubernetes.io/projected/e90c184c-0553-4d95-a020-8d499b159879-kube-api-access-2hq4p\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439585 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439550    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-home\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-home\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439728 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439590    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-tmp-dir\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439728 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439624    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90c184c-0553-4d95-a020-8d499b159879-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90c184c-0553-4d95-a020-8d499b159879-cabundle-cert\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439728 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439665    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-kserve-provision-location\") pod \"e90c184c-0553-4d95-a020-8d499b159879\" (UID: \"e90c184c-0553-4d95-a020-8d499b159879\") "
Oct 30 19:43:33.439728 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.439700    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-model-cache" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.440035 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.440000    5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-model-cache\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:33.440035 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.440026    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90c184c-0553-4d95-a020-8d499b159879-cabundle-cert" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 19:43:33.440287 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.440261    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-home" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.441933 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.441911    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90c184c-0553-4d95-a020-8d499b159879-kube-api-access-2hq4p" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "kube-api-access-2hq4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:43:33.442076 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.442055    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90c184c-0553-4d95-a020-8d499b159879-tls-certs" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:43:33.442140 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.442058    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-dshm" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.457806 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.457775    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-tmp-dir" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.470475 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.470443    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-kserve-provision-location" pod "e90c184c-0553-4d95-a020-8d499b159879" (UID: "e90c184c-0553-4d95-a020-8d499b159879"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:43:33.540743 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.540696    5025 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e90c184c-0553-4d95-a020-8d499b159879-cabundle-cert\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:33.540743 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.540739    5025 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-kserve-provision-location\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:33.540743 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.540753    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90c184c-0553-4d95-a020-8d499b159879-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:33.540989 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.540763    5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-dshm\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:33.540989 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.540772    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hq4p\" (UniqueName: \"kubernetes.io/projected/e90c184c-0553-4d95-a020-8d499b159879-kube-api-access-2hq4p\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:33.540989 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.540780    5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-home\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:33.540989 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:33.540789    5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90c184c-0553-4d95-a020-8d499b159879-tmp-dir\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:43:34.060887 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.060850    5025 generic.go:356] "Generic (PLEG): container finished" podID="e90c184c-0553-4d95-a020-8d499b159879" containerID="c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b" exitCode=0
Oct 30 19:43:34.061338 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.060891    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"
Oct 30 19:43:34.061338 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.060924    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" event={"ID":"e90c184c-0553-4d95-a020-8d499b159879","Type":"ContainerDied","Data":"c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b"}
Oct 30 19:43:34.061338 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.060961    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x" event={"ID":"e90c184c-0553-4d95-a020-8d499b159879","Type":"ContainerDied","Data":"7b8976e2e99bf76e95244028a467306fa3280f871a3f734703739d96655d73d6"}
Oct 30 19:43:34.061338 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.060986    5025 scope.go:122] "RemoveContainer" containerID="c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b"
Oct 30 19:43:34.070806 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.070788    5025 scope.go:122] "RemoveContainer" containerID="8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711"
Oct 30 19:43:34.079128 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.079109    5025 scope.go:122] "RemoveContainer" containerID="c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b"
Oct 30 19:43:34.079410 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:43:34.079392    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b\": container with ID starting with c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b not found: ID does not exist" containerID="c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b"
Oct 30 19:43:34.079473 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.079418    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b"} err="failed to get container status \"c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b\": rpc error: code = NotFound desc = could not find container \"c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b\": container with ID starting with c6eab7ae8abff84c945cebd692ad12df2f839555c9ac2943d811214608edf15b not found: ID does not exist"
Oct 30 19:43:34.079473 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.079436    5025 scope.go:122] "RemoveContainer" containerID="8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711"
Oct 30 19:43:34.079679 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:43:34.079660    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711\": container with ID starting with 8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711 not found: ID does not exist" containerID="8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711"
Oct 30 19:43:34.079728 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.079686    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711"} err="failed to get container status \"8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711\": rpc error: code = NotFound desc = could not find container \"8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711\": container with ID starting with 8b415d3292dc502781d4f41d28c0caf0c786d1f846de4ea0b8f5a8ee693fd711 not found: ID does not exist"
Oct 30 19:43:34.271028 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.270994    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"]
Oct 30 19:43:34.318099 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:34.318021    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-prefill-769794f89-qc44x"]
Oct 30 19:43:35.742753 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:35.742715    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:36.056190 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:36.056107    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90c184c-0553-4d95-a020-8d499b159879" path="/var/lib/kubelet/pods/e90c184c-0553-4d95-a020-8d499b159879/volumes"
Oct 30 19:43:45.742486 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:45.742392    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:45.742990 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:45.742509    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:43:55.742404 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:43:55.742361    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:43:58.081398 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:43:58.081365    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:44:02.939046 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:02.939019    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:44:02.990454 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:02.990399    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-kube-api-access-hkfll\" (UniqueName: \"kubernetes.io/projected/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-kube-api-access-hkfll\") pod \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") "
Oct 30 19:44:02.990570 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:02.990555    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-tls-certs\" (UniqueName: \"kubernetes.io/secret/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-tls-certs\") pod \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\" (UID: \"81858cf0-3751-4e2d-8dfc-1e93ec4b1927\") "
Oct 30 19:44:02.992586 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:02.992559    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-kube-api-access-hkfll" pod "81858cf0-3751-4e2d-8dfc-1e93ec4b1927" (UID: "81858cf0-3751-4e2d-8dfc-1e93ec4b1927"). InnerVolumeSpecName "kube-api-access-hkfll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:44:02.992723 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:02.992706    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-tls-certs" pod "81858cf0-3751-4e2d-8dfc-1e93ec4b1927" (UID: "81858cf0-3751-4e2d-8dfc-1e93ec4b1927"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:44:03.092079 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.092046    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:03.092079 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.092078    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkfll\" (UniqueName: \"kubernetes.io/projected/81858cf0-3751-4e2d-8dfc-1e93ec4b1927-kube-api-access-hkfll\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:44:03.194026 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.193989    5025 generic.go:356] "Generic (PLEG): container finished" podID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerID="7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2" exitCode=0
Oct 30 19:44:03.194204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.194038    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"
Oct 30 19:44:03.194204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.194079    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" event={"ID":"81858cf0-3751-4e2d-8dfc-1e93ec4b1927","Type":"ContainerDied","Data":"7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2"}
Oct 30 19:44:03.194204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.194144    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r" event={"ID":"81858cf0-3751-4e2d-8dfc-1e93ec4b1927","Type":"ContainerDied","Data":"42c7dca4483a831337b4ac7eb94e91311d550b5bb363c3cc28c342ac0a085982"}
Oct 30 19:44:03.194204 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.194175    5025 scope.go:122] "RemoveContainer" containerID="7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2"
Oct 30 19:44:03.204621 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.204603    5025 scope.go:122] "RemoveContainer" containerID="7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2"
Oct 30 19:44:03.204864 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:44:03.204846    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2\": container with ID starting with 7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2 not found: ID does not exist" containerID="7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2"
Oct 30 19:44:03.204913 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.204870    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2"} err="failed to get container status \"7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2\": rpc error: code = NotFound desc = could not find container \"7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2\": container with ID starting with 7b5b2d3e050944ab2837c1576e3999ed253d29ee921cf69b3e2b3d0257796fc2 not found: ID does not exist"
Oct 30 19:44:03.354515 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.354429    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"]
Oct 30 19:44:03.360044 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:03.360018    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-e4cad0c3/tls-verification-test-kserve-router-scheduler-69f7c679f8-nzd6r"]
Oct 30 19:44:04.056128 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:04.056098    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" path="/var/lib/kubelet/pods/81858cf0-3751-4e2d-8dfc-1e93ec4b1927/volumes"
Oct 30 19:44:13.222801 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.222763    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"]
Oct 30 19:44:13.223202 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223164    5025 cpu_manager.go:414] "containerMap: removing container" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="storage-initializer"
Oct 30 19:44:13.223202 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223178    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="storage-initializer"
Oct 30 19:44:13.223202 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223193    5025 cpu_manager.go:414] "containerMap: removing container" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main"
Oct 30 19:44:13.223202 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223200    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223214    5025 cpu_manager.go:414] "containerMap: removing container" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="storage-initializer"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223219    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="storage-initializer"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223230    5025 cpu_manager.go:414] "containerMap: removing container" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223235    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223248    5025 cpu_manager.go:414] "containerMap: removing container" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223253    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223357    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="81858cf0-3751-4e2d-8dfc-1e93ec4b1927" containerName="main"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223367    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90c184c-0553-4d95-a020-8d499b159879" containerName="main"
Oct 30 19:44:13.223390 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.223377    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="4076d08b-adae-4c20-87f9-a20f0ff6360b" containerName="main"
Oct 30 19:44:13.226602 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.226585    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.283869 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.283835    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a923354-376d-4dff-b10a-e7dae56156c1-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.284031 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.283980    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bz7\" (UniqueName: \"kubernetes.io/projected/8a923354-376d-4dff-b10a-e7dae56156c1-kube-api-access-m4bz7\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.292523 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.292494    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-kqm92\""
Oct 30 19:44:13.326549 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.326517    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-x7hxk\""
Oct 30 19:44:13.326689 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.326663    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Oct 30 19:44:13.326810 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.326792    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Oct 30 19:44:13.326863 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.326845    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Oct 30 19:44:13.328498 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.328480    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"]
Oct 30 19:44:13.385376 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.385347    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bz7\" (UniqueName: \"kubernetes.io/projected/8a923354-376d-4dff-b10a-e7dae56156c1-kube-api-access-m4bz7\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.385663 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.385634    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a923354-376d-4dff-b10a-e7dae56156c1-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.388246 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.388227    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a923354-376d-4dff-b10a-e7dae56156c1-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.442586 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.442559    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bz7\" (UniqueName: \"kubernetes.io/projected/8a923354-376d-4dff-b10a-e7dae56156c1-kube-api-access-m4bz7\") pod \"rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.539140 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.539053    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:13.912891 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:13.912852    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"]
Oct 30 19:44:13.917396 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:44:13.917367    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a923354_376d_4dff_b10a_e7dae56156c1.slice/crio-3df4fbc084881976c783358a76a8f9ebeb6f02f84d418a200af6e4f0fe46483f WatchSource:0}: Error finding container 3df4fbc084881976c783358a76a8f9ebeb6f02f84d418a200af6e4f0fe46483f: Status 404 returned error can't find the container with id 3df4fbc084881976c783358a76a8f9ebeb6f02f84d418a200af6e4f0fe46483f
Oct 30 19:44:14.248912 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:14.248825    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" event={"ID":"8a923354-376d-4dff-b10a-e7dae56156c1","Type":"ContainerStarted","Data":"192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c"}
Oct 30 19:44:14.248912 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:14.248877    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" event={"ID":"8a923354-376d-4dff-b10a-e7dae56156c1","Type":"ContainerStarted","Data":"3df4fbc084881976c783358a76a8f9ebeb6f02f84d418a200af6e4f0fe46483f"}
Oct 30 19:44:14.249368 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:14.248956    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:14.361564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:14.361512    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" podStartSLOduration=1.361491132 podStartE2EDuration="1.361491132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:14.359576329 +0000 UTC m=+4036.440162537" watchObservedRunningTime="2026-10-30 19:44:14.361491132 +0000 UTC m=+4036.442077341"
Oct 30 19:44:28.083075 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:44:28.083039    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:44:45.256004 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:45.255977    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:44:54.427750 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:54.427706    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"]
Oct 30 19:44:55.256140 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:44:55.256103    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:44:58.082630 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:44:58.082591    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:45:05.256508 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:05.256469    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:09.447804 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:09.447697    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main" containerID="cri-o://192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c" gracePeriod=45
Oct 30 19:45:15.256543 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:15.256503    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:15.256937 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:15.256638    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:45:25.255956 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:25.255919    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:28.083152 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:45:28.083107    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:45:35.256335 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:35.256280    5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 19:45:39.702739 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.702712    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:45:39.707329 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.707251    5025 generic.go:356] "Generic (PLEG): container finished" podID="8a923354-376d-4dff-b10a-e7dae56156c1" containerID="192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c" exitCode=0
Oct 30 19:45:39.707329 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.707283    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"
Oct 30 19:45:39.707477 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.707339    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" event={"ID":"8a923354-376d-4dff-b10a-e7dae56156c1","Type":"ContainerDied","Data":"192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c"}
Oct 30 19:45:39.707477 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.707390    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr" event={"ID":"8a923354-376d-4dff-b10a-e7dae56156c1","Type":"ContainerDied","Data":"3df4fbc084881976c783358a76a8f9ebeb6f02f84d418a200af6e4f0fe46483f"}
Oct 30 19:45:39.707477 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.707414    5025 scope.go:122] "RemoveContainer" containerID="192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c"
Oct 30 19:45:39.718386 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.718366    5025 scope.go:122] "RemoveContainer" containerID="192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c"
Oct 30 19:45:39.718773 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:45:39.718746    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c\": container with ID starting with 192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c not found: ID does not exist" containerID="192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c"
Oct 30 19:45:39.718851 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.718785    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c"} err="failed to get container status \"192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c\": rpc error: code = NotFound desc = could not find container \"192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c\": container with ID starting with 192be9cd11a406813819f76d550750e875d74f9d45aa1652244393f17aa9ef8c not found: ID does not exist"
Oct 30 19:45:39.802668 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.802632    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a923354-376d-4dff-b10a-e7dae56156c1-kube-api-access-m4bz7\" (UniqueName: \"kubernetes.io/projected/8a923354-376d-4dff-b10a-e7dae56156c1-kube-api-access-m4bz7\") pod \"8a923354-376d-4dff-b10a-e7dae56156c1\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") "
Oct 30 19:45:39.802815 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.802733    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a923354-376d-4dff-b10a-e7dae56156c1-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a923354-376d-4dff-b10a-e7dae56156c1-tls-certs\") pod \"8a923354-376d-4dff-b10a-e7dae56156c1\" (UID: \"8a923354-376d-4dff-b10a-e7dae56156c1\") "
Oct 30 19:45:39.804907 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.804877    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a923354-376d-4dff-b10a-e7dae56156c1-tls-certs" pod "8a923354-376d-4dff-b10a-e7dae56156c1" (UID: "8a923354-376d-4dff-b10a-e7dae56156c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 19:45:39.805032 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.804950    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a923354-376d-4dff-b10a-e7dae56156c1-kube-api-access-m4bz7" pod "8a923354-376d-4dff-b10a-e7dae56156c1" (UID: "8a923354-376d-4dff-b10a-e7dae56156c1"). InnerVolumeSpecName "kube-api-access-m4bz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:45:39.904138 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.904109    5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a923354-376d-4dff-b10a-e7dae56156c1-tls-certs\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:39.904138 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:39.904136    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bz7\" (UniqueName: \"kubernetes.io/projected/8a923354-376d-4dff-b10a-e7dae56156c1-kube-api-access-m4bz7\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:45:40.126003 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:40.125969    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"]
Oct 30 19:45:40.131071 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:40.131044    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-7455cbc656-d6vkr"]
Oct 30 19:45:42.055968 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:42.055924    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" path="/var/lib/kubelet/pods/8a923354-376d-4dff-b10a-e7dae56156c1/volumes"
Oct 30 19:45:49.319004 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.318962    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-72qxx/must-gather-dk7w6"]
Oct 30 19:45:49.319564 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.319544    5025 cpu_manager.go:414] "containerMap: removing container" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main"
Oct 30 19:45:49.319650 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.319567    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main"
Oct 30 19:45:49.319708 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.319670    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a923354-376d-4dff-b10a-e7dae56156c1" containerName="main"
Oct 30 19:45:49.324369 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.324348    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:49.376377 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.376341    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-72qxx\"/\"openshift-service-ca.crt\""
Oct 30 19:45:49.376553 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.376533    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-72qxx\"/\"kube-root-ca.crt\""
Oct 30 19:45:49.422865 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.422834    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-72qxx\"/\"default-dockercfg-5vcsk\""
Oct 30 19:45:49.489888 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.489847    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-must-gather-output\") pod \"must-gather-dk7w6\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") " pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:49.490062 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.489945    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28pcf\" (UniqueName: \"kubernetes.io/projected/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-kube-api-access-28pcf\") pod \"must-gather-dk7w6\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") " pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:49.511484 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.511451    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-72qxx/must-gather-dk7w6"]
Oct 30 19:45:49.591488 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.591399    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-must-gather-output\") pod \"must-gather-dk7w6\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") " pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:49.591488 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.591466    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28pcf\" (UniqueName: \"kubernetes.io/projected/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-kube-api-access-28pcf\") pod \"must-gather-dk7w6\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") " pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:49.591830 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.591805    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-must-gather-output\") pod \"must-gather-dk7w6\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") " pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:49.647706 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.647667    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28pcf\" (UniqueName: \"kubernetes.io/projected/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-kube-api-access-28pcf\") pod \"must-gather-dk7w6\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") " pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:49.937805 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:49.937770    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:45:50.308866 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:50.308787    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-72qxx/must-gather-dk7w6"]
Oct 30 19:45:50.314882 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:45:50.314851    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffe5c95e_d717_4d6a_ba50_bfb4bee8d2b2.slice/crio-a1ca0de481672565962545d1e82918589663dbbb296eccd809c1a29102e5b513 WatchSource:0}: Error finding container a1ca0de481672565962545d1e82918589663dbbb296eccd809c1a29102e5b513: Status 404 returned error can't find the container with id a1ca0de481672565962545d1e82918589663dbbb296eccd809c1a29102e5b513
Oct 30 19:45:50.316736 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:50.316715    5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 19:45:50.760927 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:50.760887    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-72qxx/must-gather-dk7w6" event={"ID":"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2","Type":"ContainerStarted","Data":"a1ca0de481672565962545d1e82918589663dbbb296eccd809c1a29102e5b513"}
Oct 30 19:45:58.087701 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:45:58.087639    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:45:58.808209 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:58.808142    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-72qxx/must-gather-dk7w6" event={"ID":"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2","Type":"ContainerStarted","Data":"ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2"}
Oct 30 19:45:58.808209 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:58.808187    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-72qxx/must-gather-dk7w6" event={"ID":"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2","Type":"ContainerStarted","Data":"cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6"}
Oct 30 19:45:58.918517 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:45:58.918464    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-72qxx/must-gather-dk7w6" podStartSLOduration=1.693873975 podStartE2EDuration="9.918451309s" totalImagesPullingTime="8.224577334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:58.918094457 +0000 UTC m=+4140.998680666" watchObservedRunningTime="2026-10-30 19:45:58.918451309 +0000 UTC m=+4140.999037515"
Oct 30 19:46:28.082446 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:46:28.082412    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:46:58.082882 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:46:58.082803    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:46:58.588836 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:46:58.588809    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:46:58.596173 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:46:58.596144    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-sfn74_b9790e2f-d93d-47f4-aa83-6c8a54567513/console-operator/2.log"
Oct 30 19:47:03.355826 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:03.355788    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-2sqns_137b793a-25e8-4205-bedb-498b6b1246d1/istio-proxy/0.log"
Oct 30 19:47:04.507753 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:04.507695    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-gtvkd_b2e16b4b-18e8-45aa-a967-185abc5a52ca/istio-proxy/0.log"
Oct 30 19:47:05.138784 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:05.138754    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-ztv2x_25f5514c-72f3-4027-b661-8cad46b22185/istio-proxy/0.log"
Oct 30 19:47:05.349164 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:05.349134    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b457d476c-cmkxg_41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/router/0.log"
Oct 30 19:47:09.091248 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:09.091220    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-2sqns_137b793a-25e8-4205-bedb-498b6b1246d1/istio-proxy/0.log"
Oct 30 19:47:10.185874 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:10.185845    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-gtvkd_b2e16b4b-18e8-45aa-a967-185abc5a52ca/istio-proxy/0.log"
Oct 30 19:47:10.857017 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:10.856985    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-5469fc87dd-ztv2x_25f5514c-72f3-4027-b661-8cad46b22185/istio-proxy/0.log"
Oct 30 19:47:11.075903 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:11.075874    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b457d476c-cmkxg_41314e0f-aadd-4cdf-aac1-93fb90cb9a5f/router/0.log"
Oct 30 19:47:14.912053 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:14.912013    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-2b7bx_e3398bb6-dec1-4048-8be9-11b71e740591/manager/0.log"
Oct 30 19:47:15.116678 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:15.116651    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6h94n_0f4a57e2-d306-4708-92d8-8d018614ebfc/kuadrant-console-plugin/0.log"
Oct 30 19:47:18.178194 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:18.178155    5025 generic.go:356] "Generic (PLEG): container finished" podID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerID="cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6" exitCode=0
Oct 30 19:47:18.178695 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:18.178206    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-72qxx/must-gather-dk7w6" event={"ID":"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2","Type":"ContainerDied","Data":"cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6"}
Oct 30 19:47:18.178695 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:18.178563    5025 scope.go:122] "RemoveContainer" containerID="cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6"
Oct 30 19:47:19.203060 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:19.203029    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-72qxx_must-gather-dk7w6_ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2/gather/0.log"
Oct 30 19:47:25.677064 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:25.677025    5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-72qxx/must-gather-dk7w6"]
Oct 30 19:47:25.677521 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:25.677226    5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-72qxx/must-gather-dk7w6" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerName="copy" containerID="cri-o://ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2" gracePeriod=2
Oct 30 19:47:25.724182 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:25.724151    5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-72qxx/must-gather-dk7w6"]
Oct 30 19:47:25.725983 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:47:25.725961    5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-dk7w6\" is forbidden: User \"system:node:ip-10-0-129-88.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-72qxx\": no relationship found between node 'ip-10-0-129-88.us-west-2.compute.internal' and this object" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:47:25.968610 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:25.968585    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-72qxx_must-gather-dk7w6_ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2/copy/0.log"
Oct 30 19:47:25.971200 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:25.971179    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:47:26.019703 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:47:26.019665    5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-dk7w6\" is forbidden: User \"system:node:ip-10-0-129-88.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-72qxx\": no relationship found between node 'ip-10-0-129-88.us-west-2.compute.internal' and this object" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:47:26.083398 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.083372    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-must-gather-output\") pod \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") "
Oct 30 19:47:26.083583 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.083408    5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-kube-api-access-28pcf\" (UniqueName: \"kubernetes.io/projected/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-kube-api-access-28pcf\") pod \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\" (UID: \"ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2\") "
Oct 30 19:47:26.085732 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.085701    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-kube-api-access-28pcf" pod "ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" (UID: "ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2"). InnerVolumeSpecName "kube-api-access-28pcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 19:47:26.090307 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.090282    5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-must-gather-output" pod "ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" (UID: "ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 19:47:26.184750 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.184715    5025 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-must-gather-output\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:26.184750 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.184745    5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28pcf\" (UniqueName: \"kubernetes.io/projected/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2-kube-api-access-28pcf\") on node \"ip-10-0-129-88.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 19:47:26.215697 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.215624    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-72qxx_must-gather-dk7w6_ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2/copy/0.log"
Oct 30 19:47:26.216026 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.216001    5025 generic.go:356] "Generic (PLEG): container finished" podID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerID="ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2" exitCode=143
Oct 30 19:47:26.216077 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.216052    5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-72qxx/must-gather-dk7w6"
Oct 30 19:47:26.216122 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.216102    5025 scope.go:122] "RemoveContainer" containerID="ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2"
Oct 30 19:47:26.227664 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.227643    5025 scope.go:122] "RemoveContainer" containerID="cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6"
Oct 30 19:47:26.240023 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.240006    5025 scope.go:122] "RemoveContainer" containerID="ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2"
Oct 30 19:47:26.240310 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:47:26.240291    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2\": container with ID starting with ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2 not found: ID does not exist" containerID="ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2"
Oct 30 19:47:26.240429 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.240340    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2"} err="failed to get container status \"ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2\": rpc error: code = NotFound desc = could not find container \"ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2\": container with ID starting with ac9bfe78d5286273e5bfa94bfa6c77a96169cd9579d7c41e51ca0f6b580204d2 not found: ID does not exist"
Oct 30 19:47:26.240429 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.240362    5025 scope.go:122] "RemoveContainer" containerID="cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6"
Oct 30 19:47:26.240680 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:47:26.240651    5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6\": container with ID starting with cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6 not found: ID does not exist" containerID="cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6"
Oct 30 19:47:26.240724 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:26.240690    5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6"} err="failed to get container status \"cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6\": rpc error: code = NotFound desc = could not find container \"cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6\": container with ID starting with cbcbe8e65f2f7d0b94389cf597e6142610fbc403563b65a5253fba6f31b608a6 not found: ID does not exist"
Oct 30 19:47:28.056080 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:28.056041    5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" path="/var/lib/kubelet/pods/ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2/volumes"
Oct 30 19:47:28.082864 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:47:28.082831    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:47:30.415152 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:30.415120    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k8tk8_daacb5ca-b3fa-4344-add9-c451fc1e5e25/global-pull-secret-syncer/0.log"
Oct 30 19:47:31.944673 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:31.944640    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-8cw5s_4a159864-3010-4195-84de-1407b16eb5c2/connection-checker/0.log"
Oct 30 19:47:32.381753 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:32.381724    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rw5pb_1e6a72c5-3683-4fc8-8525-e82baed6a4ad/connection-checker/0.log"
Oct 30 19:47:32.825367 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:32.825270    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-vvw8v_a9c1a5cc-0e5f-46ef-bc0a-e405fd732eeb/connection-checker/0.log"
Oct 30 19:47:33.846711 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:33.846680    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5b6w7_511cf18c-8d23-40e7-9e68-f683e3fe9e7d/konnectivity-agent/0.log"
Oct 30 19:47:35.630381 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:35.630353    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-88.us-west-2.compute.internal_82007190ff1dd353d8af659976a136ef/haproxy/0.log"
Oct 30 19:47:40.552634 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:40.552606    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-68df894cb4-2b7bx_e3398bb6-dec1-4048-8be9-11b71e740591/manager/0.log"
Oct 30 19:47:40.950654 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:40.950627    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6h94n_0f4a57e2-d306-4708-92d8-8d018614ebfc/kuadrant-console-plugin/0.log"
Oct 30 19:47:42.494807 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.494776    5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"]
Oct 30 19:47:42.495196 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.495157    5025 cpu_manager.go:414] "containerMap: removing container" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerName="gather"
Oct 30 19:47:42.495196 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.495169    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerName="gather"
Oct 30 19:47:42.495196 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.495184    5025 cpu_manager.go:414] "containerMap: removing container" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerName="copy"
Oct 30 19:47:42.495196 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.495189    5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerName="copy"
Oct 30 19:47:42.495348 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.495281    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerName="gather"
Oct 30 19:47:42.495348 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.495290    5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe5c95e-d717-4d6a-ba50-bfb4bee8d2b2" containerName="copy"
Oct 30 19:47:42.498540 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.498523    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.523229 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.523208    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-proc\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.523353 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.523254    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-lib-modules\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.523429 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.523380    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-sys\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.523498 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.523478    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/69ea7ce4-f052-4695-9abd-e59af4c313ec-kube-api-access-zb5nq\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.523547 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.523509    5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-podres\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.548995 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.548967    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ndc8\"/\"openshift-service-ca.crt\""
Oct 30 19:47:42.596516 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.596479    5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5ndc8\"/\"kube-root-ca.crt\""
Oct 30 19:47:42.596657 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.596544    5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5ndc8\"/\"default-dockercfg-xx74q\""
Oct 30 19:47:42.599013 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.598990    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"]
Oct 30 19:47:42.624291 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624266    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-proc\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624306    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-lib-modules\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624372    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-sys\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624413 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624392    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-proc\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624418    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-lib-modules\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624447    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-sys\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624455    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/69ea7ce4-f052-4695-9abd-e59af4c313ec-kube-api-access-zb5nq\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624526 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624480    5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-podres\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.624651 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.624575    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/69ea7ce4-f052-4695-9abd-e59af4c313ec-podres\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.680777 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.680751    5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb5nq\" (UniqueName: \"kubernetes.io/projected/69ea7ce4-f052-4695-9abd-e59af4c313ec-kube-api-access-zb5nq\") pod \"perf-node-gather-daemonset-9hvdg\" (UID: \"69ea7ce4-f052-4695-9abd-e59af4c313ec\") " pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:42.811207 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:42.811135    5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:43.177671 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:43.177637    5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"]
Oct 30 19:47:43.183269 ip-10-0-129-88 kubenswrapper[5025]: W1030 19:47:43.183244    5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod69ea7ce4_f052_4695_9abd_e59af4c313ec.slice/crio-a2b99b93868ac5e259db6fcebad1af6d46888673262cceaa3c4dbd5e88c04dd4 WatchSource:0}: Error finding container a2b99b93868ac5e259db6fcebad1af6d46888673262cceaa3c4dbd5e88c04dd4: Status 404 returned error can't find the container with id a2b99b93868ac5e259db6fcebad1af6d46888673262cceaa3c4dbd5e88c04dd4
Oct 30 19:47:43.298531 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:43.298495    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg" event={"ID":"69ea7ce4-f052-4695-9abd-e59af4c313ec","Type":"ContainerStarted","Data":"828a00a83eded43d1c16e7b42bfe20c72c203128fea6056d9790453595cfb259"}
Oct 30 19:47:43.298670 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:43.298554    5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg" event={"ID":"69ea7ce4-f052-4695-9abd-e59af4c313ec","Type":"ContainerStarted","Data":"a2b99b93868ac5e259db6fcebad1af6d46888673262cceaa3c4dbd5e88c04dd4"}
Oct 30 19:47:43.298670 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:43.298585    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:43.408260 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:43.408210    5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg" podStartSLOduration=1.4081938680000001 podStartE2EDuration="1.408193868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:43.407571324 +0000 UTC m=+4245.488157529" watchObservedRunningTime="2026-10-30 19:47:43.408193868 +0000 UTC m=+4245.488780074"
Oct 30 19:47:47.486466 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:47.486433    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-4gt4s_e487a2e3-2c52-4dea-80a6-fdb09e9229ae/cluster-monitoring-operator/0.log"
Oct 30 19:47:49.314113 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:49.314080    5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5ndc8/perf-node-gather-daemonset-9hvdg"
Oct 30 19:47:51.104090 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:51.104059    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8nhc_14b90994-57e8-4f52-9751-20a02c662a2c/node-exporter/0.log"
Oct 30 19:47:51.510240 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:51.510210    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8nhc_14b90994-57e8-4f52-9751-20a02c662a2c/kube-rbac-proxy/0.log"
Oct 30 19:47:51.904837 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:51.904810    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r8nhc_14b90994-57e8-4f52-9751-20a02c662a2c/init-textfile/0.log"
Oct 30 19:47:57.416540 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:57.416509    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-48dr5_cb07387c-b9fd-4592-beef-0264e73732d6/prometheus-operator/0.log"
Oct 30 19:47:57.808956 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:57.808865    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-48dr5_cb07387c-b9fd-4592-beef-0264e73732d6/kube-rbac-proxy/0.log"
Oct 30 19:47:58.082022 ip-10-0-129-88 kubenswrapper[5025]: E1030 19:47:58.081942    5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d52ch" containerName="extractor"
Oct 30 19:47:58.205892 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:47:58.205867    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-fhbbl_72dd3f31-31ed-41a4-a097-8d01f4063960/prometheus-operator-admission-webhook/0.log"
Oct 30 19:48:04.455508 ip-10-0-129-88 kubenswrapper[5025]: I1030 19:48:04.455481    5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-s4bss_334cba69-4d11-4125-9763-d79d806afbc8/networking-console-plugin/0.log"