Oct 30 02:54:18.838513 ip-10-0-138-200 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:54:19.277872 ip-10-0-138-200 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:19.277872 ip-10-0-138-200 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:54:19.277872 ip-10-0-138-200 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:19.277872 ip-10-0-138-200 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:54:19.284373 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284282    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.284373 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284369    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.284373 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284373    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.284373 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284377    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.284373 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284380    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284384    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284387    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284390    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284393    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284396    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284402    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284406    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284408    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284411    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284413    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284416    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284419    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284422    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284427    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284431    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284435    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284439    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284445    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.284566 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284448    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284453    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284458    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284462    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284465    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284467    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284470    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284473    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284475    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284478    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284481    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284484    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284489    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284492    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284494    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284500    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284503    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284506    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284509    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.285013 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284513    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284515    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284518    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284521    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284523    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284528    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284531    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284534    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284537    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284540    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284542    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284545    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284548    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284551    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284554    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284556    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284559    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284561    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284566    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284569    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.285471 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284572    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284575    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284578    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284581    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284583    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284586    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284589    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284591    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284594    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284598    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284600    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284605    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284608    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284611    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284613    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284616    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284618    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284621    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284624    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284626    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.285931 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284629    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284631    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284634    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284639    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284642    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284645    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284648    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284651    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284653    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284656    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284659    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284661    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284664    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284667    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284669    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284672    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284676    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284679    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284682    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.286411 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284684    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.286851 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284688    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.286851 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284693    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.286851 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284695    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.286851 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.284698    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.287233 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287219    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.287233 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287233    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287238    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287241    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287245    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287249    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287252    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287255    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287258    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287260    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287265    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287268    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287270    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287273    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287278    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287281    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287283    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287286    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287289    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287291    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287294    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.287294 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287297    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287299    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287304    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287307    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287312    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287314    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287320    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287327    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287330    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287335    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287340    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287343    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287345    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287349    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287352    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287356    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287359    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287362    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287364    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.287773 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287367    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287369    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287372    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287374    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287377    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287379    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287382    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287385    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287387    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287390    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287392    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287395    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287397    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287400    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287402    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287406    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287410    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287412    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287415    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.288259 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287418    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287421    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287423    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287426    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287428    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287431    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287434    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287436    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287440    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287444    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287447    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287449    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287452    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287454    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287457    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287460    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287463    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287466    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287469    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287472    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.288722 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287475    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287477    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287480    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287482    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287485    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287487    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287490    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287493    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287496    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287498    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287501    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287504    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287506    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287510    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287512    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287515    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287517    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287521    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287523    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287526    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.289221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287528    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287531    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287533    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287536    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287538    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287541    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.287544    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288073    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288079    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288083    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288086    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288089    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288091    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288095    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288097    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288100    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288103    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288105    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288108    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288110    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.289687 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288113    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288116    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288119    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288121    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288124    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288126    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288129    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288132    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288134    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288137    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288139    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288142    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288144    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288147    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288150    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288152    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288168    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288171    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288174    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288177    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.290166 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288180    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288183    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288186    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288189    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288191    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288194    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288197    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288200    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288202    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288205    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288207    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288210    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288213    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288215    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288218    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288220    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288223    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288225    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288228    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.290643 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288230    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288233    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288235    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288238    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288240    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288243    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288246    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288248    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288251    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288254    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288256    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288259    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288262    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288264    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288267    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288270    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288274    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288277    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288279    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.291225 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288282    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288285    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288288    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288290    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288293    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288295    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288298    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288300    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288304    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288308    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288311    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288314    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288316    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288318    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288321    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288324    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288326    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288329    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288331    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.291677 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288334    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288337    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288341    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288344    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288347    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288353    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288356    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288359    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288362    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288364    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288367    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288371    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288374    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288377    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288380    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.288382    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289428    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289438    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289445    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289449    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289454    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:54:19.292126 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289457    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289465    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289469    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289473    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289476    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289480    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289483    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289486    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289489    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289492    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289495    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289498    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289501    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289507    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289510    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289513    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289518    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289522    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289526    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289529    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289533    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289536    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289540    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289544    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:54:19.292679 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289547    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289550    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289553    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289562    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289565    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289568    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289571    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289574    2574 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289577    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289582    2574 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289585    2574 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289588    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289592    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289595    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289599    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289602    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289605    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289608    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289611    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289614    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289617    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289620    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289625    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289628    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289632    2574 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:54:19.293296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289636    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289640    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289643    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289647    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289650    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289653    2574 flags.go:64] FLAG: --help="false"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289656    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289660    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289663    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289667    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289670    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289674    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289677    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289680    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289682    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289686    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289689    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289692    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289695    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289698    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289701    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289705    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289707    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289710    2574 flags.go:64] FLAG: --lock-file=""
Oct 30 02:54:19.293901 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289713    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289717    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289720    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289726    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289728    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289732    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289736    2574 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289739    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289743    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289746    2574 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289749    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289753    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289756    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289760    2574 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289763    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289766    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289772    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289775    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289778    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289781    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289784    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289791    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289794    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289798    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289801    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:54:19.294494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289804    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289807    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289810    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289813    2574 flags.go:64] FLAG: --port="10250"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289816    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289819    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0006fa8e21ebc9997"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289822    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289825    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289828    2574 flags.go:64] FLAG: --register-node="true"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289831    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289834    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289837    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289840    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289843    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289847    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289850    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289853    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289856    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289859    2574 flags.go:64] FLAG: --runonce="false"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289862    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289866    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289869    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289872    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289875    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289878    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289882    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:54:19.295103 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289885    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289889    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289892    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289895    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289898    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289901    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289904    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289907    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289912    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289915    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289918    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289922    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289925    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289928    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289931    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289934    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289937    2574 flags.go:64] FLAG: --v="2"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289941    2574 flags.go:64] FLAG: --version="false"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289945    2574 flags.go:64] FLAG: --vmodule=""
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289949    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.289952    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290098    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290102    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.295813 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290107    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290111    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290114    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290118    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290120    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290124    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290128    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290131    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290138    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290141    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290144    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290146    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290149    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290152    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290167    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290170    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290173    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290176    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290178    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.296468 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290181    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290184    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290186    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290189    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290191    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290194    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290196    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290199    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290201    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290205    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290207    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290210    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290213    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290216    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290218    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290221    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290223    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290226    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290228    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290231    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.296913 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290234    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290236    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290244    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290247    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290250    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290252    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290255    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290259    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290262    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290264    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290267    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290269    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290272    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290274    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290277    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290280    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290283    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290285    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290288    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.297428 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290291    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290295    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290297    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290300    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290302    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290305    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290308    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290310    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290313    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290315    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290318    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290320    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290342    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290346    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290350    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290353    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290357    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290361    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290363    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290366    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.297897 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290370    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290373    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290375    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290378    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290380    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290383    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290385    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290388    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290391    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290394    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290396    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290399    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290402    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290406    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290408    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290411    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290414    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290416    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290419    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290421    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.298395 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290424    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290426    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290429    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290431    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290434    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290437    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.290465    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290502    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290506    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290509    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290513    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290515    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290518    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290522    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290524    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290527    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.298865 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290530    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290532    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290535    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290537    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290540    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290543    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290545    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290548    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290551    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290554    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290557    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290560    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290562    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290565    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290568    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290570    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290573    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290576    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290578    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290580    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.299262 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290583    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290586    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290588    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290591    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290593    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290596    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290599    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290602    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290604    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290608    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290611    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290613    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290616    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290618    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290621    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290623    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290626    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290629    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290632    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.299732 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290634    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290637    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290640    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290643    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290646    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290648    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290651    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290654    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290656    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290659    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290661    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290664    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290666    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290669    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290672    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290674    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290677    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290679    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290682    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290684    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.300270 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290687    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290690    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290694    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290696    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290700    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290703    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290706    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290708    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290711    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290714    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290716    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290719    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290721    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290724    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290727    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290730    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290733    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290735    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290738    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290741    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.300740 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290743    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290746    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290748    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290751    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290753    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290756    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290758    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290761    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290763    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290766    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290770    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290773    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290776    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290780    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290786    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290788    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290792    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.290795    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.301221 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.297848    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.297864    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.298611    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.298619    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298677    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298682    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298686    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298690    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298692    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298695    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298698    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298701    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298703    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298706    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298708    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298711    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298714    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298717    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298719    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.301648 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298722    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298725    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298727    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298730    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298732    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298735    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298738    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298740    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298743    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298745    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298748    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298751    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298754    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298757    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298761    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298764    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298767    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298769    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298774    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298777    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.302097 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298781    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298784    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298787    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298790    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298793    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298797    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298799    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298802    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298805    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298808    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298811    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298814    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298817    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298820    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298822    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298825    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298828    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298830    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298833    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.302612 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298836    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298839    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298841    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298844    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298847    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298850    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298853    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298855    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298859    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298862    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298865    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298867    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298870    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298873    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298876    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298878    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298881    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298884    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298886    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298889    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.303122 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298892    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298894    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298897    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298899    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298902    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298905    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298908    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298911    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298914    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298916    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298919    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298921    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298926    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298929    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298933    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298935    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298938    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298941    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298944    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.303604 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298946    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298949    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298952    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298955    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298957    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298960    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298962    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298965    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298967    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298970    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298972    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298975    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.298977    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.299006    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299148    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299169    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.304054 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299174    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299176    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299179    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299182    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299185    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299188    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299191    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299194    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299196    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299199    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299202    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299205    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299208    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299210    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299213    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299216    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299218    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299221    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299224    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299227    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.304454 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299230    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299233    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299236    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299238    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299241    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299244    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299247    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299249    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299252    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299254    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299257    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299259    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299262    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299264    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299267    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299270    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299272    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299275    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299278    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299280    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.304923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299283    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299287    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299290    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299293    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299295    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299298    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299301    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299303    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299306    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299308    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299311    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299313    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299316    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299319    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299322    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299324    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299327    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299329    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299332    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299335    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.305429 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299337    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299340    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299343    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299345    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299348    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299350    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299353    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299356    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299358    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299361    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299364    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299366    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299369    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299372    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299374    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299377    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299379    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299382    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299385    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299387    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.305923 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299390    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299392    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299395    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299397    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299400    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299403    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299405    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299409    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299414    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299417    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299421    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299425    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299429    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299432    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299435    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299437    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299440    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299443    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.306416 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299445    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299448    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299451    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299453    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299456    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299458    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.299483    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299516    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299520    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299523    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299526    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299529    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299532    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299535    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299538    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:54:19.306850 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299540    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299544    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299548    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299551    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299555    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299560    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299563    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299567    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299570    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299572    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299575    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299578    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299580    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299583    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299586    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299588    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299591    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299593    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:54:19.307286 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299596    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299598    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299601    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299604    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299606    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299609    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299612    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299614    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299616    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299619    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299622    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299624    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299627    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299629    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299632    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299634    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299637    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299640    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299642    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299645    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:54:19.307731 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299648    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299650    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299653    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299655    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299658    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299661    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299663    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299666    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299669    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299672    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299675    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299677    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299680    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299682    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299685    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299687    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299690    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299692    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299695    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:54:19.308254 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299697    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299700    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299702    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299705    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299707    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299710    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299712    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299715    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299717    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299720    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299722    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299725    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299727    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299733    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299737    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299742    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299745    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299748    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299750    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299753    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:54:19.308712 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299755    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299758    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299760    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299763    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299766    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299768    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299771    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299773    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299776    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299779    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299781    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299784    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299786    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299789    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299792    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299794    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299797    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299799    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299801    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299804    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:54:19.309288 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:19.299807    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:54:19.309770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.299870    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:54:19.309770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.302240    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 02:54:19.309770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.303306    2574 server.go:1014] "Starting client certificate rotation"
Oct 30 02:54:19.309770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.303410    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:19.309770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.304267    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:54:19.309770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.305519    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:19.309770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.307010    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:54:19.320377 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.320356    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:54:19.324384 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.324370    2574 log.go:25] "Validated CRI v1 image API"
Oct 30 02:54:19.325319 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.325295    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:54:19.327976 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.327943    2574 fs.go:135] Filesystem UUIDs: map[3142cb85-9a18-4850-b4c5-9f789ec9e743:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ae4c1181-9583-4106-bead-23f259c85786:/dev/nvme0n1p4]
Oct 30 02:54:19.327976 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.327971    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 02:54:19.334129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.334020    2574 manager.go:217] Machine: {Timestamp:2026-10-30 02:54:19.332589493 +0000 UTC m=+0.375702346 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098901 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a3e4fb1f3aeee7ce3c1742ee15fdf SystemUUID:ec2a3e4f-b1f3-aeee-7ce3-c1742ee15fdf BootID:687a6ca5-73a4-4842-acd3-4ae67797a993 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:51:c5:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:51:c5:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:ed:cd:a0:42:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:54:19.334129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.334128    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:54:19.334250 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.334229    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:54:19.337760 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.337729    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:54:19.337903 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.337762    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:54:19.338868 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.338857    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:54:19.338903 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.338870    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:54:19.338903 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.338899    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:19.338953 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.338913    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:54:19.340418 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.340407    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:54:19.345696 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.345683    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:54:19.345817 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.345808    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:54:19.346661 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.346651    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:54:19.346702 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.346664    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:54:19.346702 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.346675    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:54:19.346702 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.346685    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:54:19.346702 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.346693    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:54:19.347060 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.347046    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:19.349741 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.349725    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:54:19.356422 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.356409    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:54:19.357363 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357351    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:54:19.357399 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357381    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:54:19.357529 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357520    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:54:19.357563 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357535    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:54:19.357563 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357542    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:54:19.357563 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357547    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:54:19.357563 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357553    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:54:19.357563 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.357559    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:54:19.358230 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.358222    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:54:19.358266 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.358232    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:54:19.358266 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.358241    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:54:19.358266 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.358247    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:54:19.358266 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.358264    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:54:19.358382 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.358274    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:54:19.359087 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.359079    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:54:19.359118 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.359088    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:54:19.363252 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.363224    2574 server.go:1263] "Started kubelet"
Oct 30 02:54:19.363738 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.363699    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:54:19.363875 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.363829    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:54:19.363936 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.363903    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:54:19.364248 ip-10-0-138-200 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:54:19.365507 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.365483    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:54:19.367429 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.367414    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:54:19.373051 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.373027    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 02:54:19.373570 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.373552    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:19.374215 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.374201    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:54:19.374380 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.374359    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:19.374930 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.374914    2574 factory.go:55] Registering systemd factory
Oct 30 02:54:19.374930 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.374933    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:54:19.375121 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375107    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:54:19.375121 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375111    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:54:19.375240 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375130    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:54:19.375240 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375123    2574 factory.go:153] Registering CRI-O factory
Oct 30 02:54:19.375240 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375183    2574 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:54:19.375240 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375220    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:54:19.375240 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375228    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:54:19.375240 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375234    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:54:19.375487 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375250    2574 factory.go:103] Registering Raw factory
Oct 30 02:54:19.375487 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.375261    2574 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:54:19.376446 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.376433    2574 manager.go:319] Starting recovery of all containers
Oct 30 02:54:19.387594 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.387575    2574 manager.go:324] Recovery completed
Oct 30 02:54:19.392311 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.392299    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:19.393907 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.393891    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:19.393970 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.393921    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:19.393970 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.393942    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:19.394463 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.394450    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:54:19.394463 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.394461    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:54:19.394592 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.394480    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:54:19.394592 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.394516    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:19.395737 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.395714    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:19.395815 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.395770    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:19.395815 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.395788    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:19.396642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.396629    2574 policy_none.go:50] "Start"
Oct 30 02:54:19.396642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.396645    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:54:19.396747 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.396656    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:54:19.398452 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.398407    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-200.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 02:54:19.398675 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.398651    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 02:54:19.398768 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.398749    2574 policy_none.go:44] "Start"
Oct 30 02:54:19.399973 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.398416    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-200.ec2.internal.18e32f189bcc7ec4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-200.ec2.internal,},FirstTimestamp:2026-10-30 02:54:19.363180228 +0000 UTC m=+0.406293086,LastTimestamp:2026-10-30 02:54:19.363180228 +0000 UTC m=+0.406293086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-200.ec2.internal,}"
Oct 30 02:54:19.401073 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.401052    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:19.407242 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.407226    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sshvc"
Oct 30 02:54:19.419470 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.419448    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sshvc"
Oct 30 02:54:19.420409 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.420282    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-200.ec2.internal.18e32f189da155a4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-200.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-200.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-200.ec2.internal,},FirstTimestamp:2026-10-30 02:54:19.393906084 +0000 UTC m=+0.437018936,LastTimestamp:2026-10-30 02:54:19.393906084 +0000 UTC m=+0.437018936,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-200.ec2.internal,}"
Oct 30 02:54:19.420528 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.420429    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 02:54:19.420528 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.420358    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-200.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 02:54:19.435281 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.435248    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:54:19.439838 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.439821    2574 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:54:19.439922 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.439856    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:54:19.439922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.439865    2574 server.go:87] "Starting device plugin registration server"
Oct 30 02:54:19.440129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.440116    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:54:19.440190 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.440131    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:54:19.440354 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.440325    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:54:19.440440 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.440399    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:54:19.440440 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.440411    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:54:19.441106 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.441086    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:54:19.441200 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.441139    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:19.477352 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.461776    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:19.511218 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.511186    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:54:19.512414 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.512399    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:54:19.512496 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.512424    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:54:19.512496 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.512446    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:54:19.512496 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.512487    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 02:54:19.525113 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.525096    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:19.540986 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.540806    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:19.541900 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.541883    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:19.541986 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.541912    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:19.541986 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.541923    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:19.541986 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.541949    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.559792 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.559769    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.559792 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.559792    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-200.ec2.internal\": node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:19.604100 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.604073    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:19.613510 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.613487    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"]
Oct 30 02:54:19.613566 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.613546    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:19.614511 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.614491    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:19.614607 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.614520    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:19.614607 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.614529    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:19.616510 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.616496    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:19.616658 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.616645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.616698 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.616673    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:19.617250 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.617232    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:19.617337 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.617249    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:19.617337 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.617262    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:19.617337 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.617270    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:19.617337 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.617279    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:19.617458 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.617280    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:19.619698 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.619683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.619793 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.619714    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:19.620523 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.620506    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:19.620592 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.620540    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:19.620592 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.620555    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:19.657523 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.657500    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-200.ec2.internal\" not found" node="ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.661223 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.661199    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-200.ec2.internal\" not found" node="ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.677028 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.677002    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.677137 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.677034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.677137 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.677058    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.704280 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.704254    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:19.777453 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.777418    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.777453 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.777457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.777632 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.777481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.777632 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.777521    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.777632 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.777525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.777632 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.777525    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.805245 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.805199    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:19.905475 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:19.905450    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:19.961692 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.961668    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:19.964692 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:19.964661    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:20.006231 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:20.006202    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:21.757042 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:21.757012    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:21.857606 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:21.857571    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:21.954074 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:21.954041    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 02:54:21.954579 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:21.954212    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 02:54:21.957938 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:21.957922    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:22.058743 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:22.058708    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:22.075771 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.075718    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:21 +0000 UTC" deadline="2028-05-30 12:55:07.950755114 +0000 UTC"
Oct 30 02:54:22.075771 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.075766    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13882h0m45.874991951s"
Oct 30 02:54:22.129206 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:22.129172    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d685204da0cc5fa69ed201b4cfe0a4.slice/crio-a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9 WatchSource:0}: Error finding container a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9: Status 404 returned error can't find the container with id a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9
Oct 30 02:54:22.138896 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.138876    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:54:22.146163 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:22.146143    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd03e27bf6704eb151f6eb79c368f447.slice/crio-0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3 WatchSource:0}: Error finding container 0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3: Status 404 returned error can't find the container with id 0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3
Oct 30 02:54:22.159378 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:22.159358    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:22.166074 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.166022    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3"}
Oct 30 02:54:22.167016 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.166997    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9"}
Oct 30 02:54:22.251898 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.251863    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:54:22.253149 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.253129    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:54:22.253266 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.253161    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:54:22.253266 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.253173    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:54:22.260383 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:22.260366    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:22.360791 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:22.360728    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:54:22.383852 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.383835    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:22.424951 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.424903    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:22.450982 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.450957    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:22.451093 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.451075    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:54:22.473289 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.473269    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:54:22.620561 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.620456    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:22.998435 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:22.998363    2574 apiserver.go:52] "Watching apiserver"
Oct 30 02:54:23.025107 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.025073    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 02:54:23.045429 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.045400    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:54:23.045770 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.045751    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal","openshift-network-operator/iptables-alerter-fttld","openshift-ovn-kubernetes/ovnkube-node-fvzlt","kube-system/global-pull-secret-syncer-lhtq6","openshift-multus/multus-5bwp8","openshift-multus/multus-additional-cni-plugins-6x562","openshift-multus/network-metrics-daemon-s258j","openshift-dns/node-resolver-x8h8g","openshift-network-diagnostics/network-check-target-mrnst","kube-system/konnectivity-agent-j9s8d","openshift-image-registry/node-ca-zq2m7","kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f","openshift-cluster-node-tuning-operator/tuned-mwh8j"]
Oct 30 02:54:23.048392 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.048367    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:23.048502 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.048448    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:23.050610 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.050585    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.052932 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.052912    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.055183 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.055162    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.055284 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.055243    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:23.057486 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.057468    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.060582 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.060182    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.062490 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.062474    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:23.062594 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.062544    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:23.065885 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.065399    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.070700 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.070675    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:54:23.071076 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.071051    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:54:23.071753 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.071656    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:54:23.071753 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.071706    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5k8rj\""
Oct 30 02:54:23.072045 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.072025    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.074930 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.074910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.077323 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077168    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cr2zp\""
Oct 30 02:54:23.077323 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077268    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sdsc6\""
Oct 30 02:54:23.077323 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077271    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:54:23.077601 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077583    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:54:23.077688 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077618    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:54:23.077756 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077748    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:54:23.077808 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077783    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:54:23.077945 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.077928    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.078123 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.078080    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:54:23.078759 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.078359    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:54:23.079054 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.078555    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 02:54:23.079464 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.078576    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:54:23.079586 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.078636    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:54:23.079668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.078651    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:54:23.079738 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.078724    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:54:23.079819 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.079015    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vxw8k\""
Oct 30 02:54:23.079892 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.079431    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:54:23.080623 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.080267    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.085998 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.085977    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:23.086637 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.086619    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8mm7k"
Oct 30 02:54:23.098866 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.098847    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8mm7k"
Oct 30 02:54:23.126747 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.126728    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:54:23.139165 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.139278 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-sys\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.139278 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139204    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqk7b\" (UniqueName: \"kubernetes.io/projected/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-kube-api-access-sqk7b\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.139278 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jh6m\" (UniqueName: \"kubernetes.io/projected/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-kube-api-access-4jh6m\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.139419 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-device-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.139419 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139340    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98nz7\" (UniqueName: \"kubernetes.io/projected/9fbf954b-1744-434a-8f0f-7ac3d546e081-kube-api-access-98nz7\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.139419 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139365    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-kubelet-config\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.139419 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139404    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-tuned\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.139599 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139423    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ef68d50-d2d2-4244-848f-208cf21b039a-konnectivity-ca\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.139599 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-socket-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.139599 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139466    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-var-lib-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.139599 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139539    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-log-socket\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.139599 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139558    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-system-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.139599 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-hostroot\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139592    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-socket-dir-parent\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139626    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139666    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysconfig\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa50191b-3520-4499-a385-a0e6a40c3da5-host-slash\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139733    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139756    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-systemd-units\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139779    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-bin\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.139816 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139804    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-multus-daemon-config\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-cnibin\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139863    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6f47\" (UniqueName: \"kubernetes.io/projected/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-kube-api-access-r6f47\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139881    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-registration-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-env-overrides\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bppxt\" (UniqueName: \"kubernetes.io/projected/27151663-70b9-4779-9730-fd65ee35bfcb-kube-api-access-bppxt\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-conf\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139965    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-host\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.139986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa50191b-3520-4499-a385-a0e6a40c3da5-iptables-alerter-script\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140013    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ef68d50-d2d2-4244-848f-208cf21b039a-agent-certs\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140082    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-ovn\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140117    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bjg\" (UniqueName: \"kubernetes.io/projected/fe685890-13f3-4526-921b-56a13e3dcea6-kube-api-access-q8bjg\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-systemd\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-tmp\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140167    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-netns\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140181    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-node-log\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140205    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140223    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27151663-70b9-4779-9730-fd65ee35bfcb-ovn-node-metrics-cert\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-script-lib\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140258    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-cni-binary-copy\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140273    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-modprobe-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140301    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-sys-fs\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140316    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-netd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-config\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140349    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-kubelet\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140363    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-multus-certs\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140384    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.140466 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140399    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-var-lib-kubelet\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140412    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-kubelet\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140426    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-slash\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-systemd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140455    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-etc-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-serviceca\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140498    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-k8s-cni-cncf-io\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140514    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140546    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-host\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-multus\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-conf-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-etc-kubernetes\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140757    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcng4\" (UniqueName: \"kubernetes.io/projected/f106e28a-9a25-4d6b-a16f-add5d5591b60-kube-api-access-hcng4\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140797    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-kubernetes\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-hosts-file\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-tmp-dir\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.141148 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-dbus\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-cnibin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.140934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-system-cni-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141030    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-run\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141080    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zdcj\" (UniqueName: \"kubernetes.io/projected/fa50191b-3520-4499-a385-a0e6a40c3da5-kube-api-access-6zdcj\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrqvb\" (UniqueName: \"kubernetes.io/projected/453941ab-5ec2-4b68-9991-492690f822be-kube-api-access-rrqvb\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141129    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-lib-modules\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-netns\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.141886 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.141172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-bin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.172538 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172499    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:54:23.172676 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172548    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:54:23.172676 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172578    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:54:23.172676 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172588    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg86w\""
Oct 30 02:54:23.172676 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172597    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4s4b\""
Oct 30 02:54:23.172676 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172612    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vnlvm\""
Oct 30 02:54:23.172676 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172607    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f9m7f\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172725    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172746    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172757    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172790    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tm8xk\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172802    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172751    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172804    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172830    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:54:23.172973 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.172837    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:54:23.241929 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.241894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-netd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.241929 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.241935    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-config\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.241958    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-kubelet\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-multus-certs\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242018    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-netd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242033    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242069    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-var-lib-kubelet\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242092    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-kubelet\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242098    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-kubelet\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-slash\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.242173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242150    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-systemd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.242642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242188    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-etc-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.242642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-serviceca\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.242642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:23.242642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-k8s-cni-cncf-io\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.242642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.242642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242323    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-host\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.242642 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-multus\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.242916 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242752    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-conf-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.242916 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.242848    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-etc-kubernetes\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.244809 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcng4\" (UniqueName: \"kubernetes.io/projected/f106e28a-9a25-4d6b-a16f-add5d5591b60-kube-api-access-hcng4\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.244809 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244607    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-systemd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.244809 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244676    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-multus\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.244809 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244730    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-kubernetes\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.244809 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-hosts-file\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.244809 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-tmp-dir\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244870    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-dbus\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244901    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.244916    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244987    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-cnibin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245053    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-var-lib-kubelet\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.245069    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:23.745020892 +0000 UTC m=+3.137770071 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245100    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-host\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244932    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-cnibin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245138    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-system-cni-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.245172 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245142    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-k8s-cni-cncf-io\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245183    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245323    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-kubelet\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245346    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245366    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-run\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245405    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdcj\" (UniqueName: \"kubernetes.io/projected/fa50191b-3520-4499-a385-a0e6a40c3da5-kube-api-access-6zdcj\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245618    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-serviceca\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245688    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-system-cni-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.245707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-kubernetes\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.245991 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245768    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-hosts-file\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.245991 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.244778    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-etc-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.245991 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245847    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.245991 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245907    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-run\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.245991 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245954    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqvb\" (UniqueName: \"kubernetes.io/projected/453941ab-5ec2-4b68-9991-492690f822be-kube-api-access-rrqvb\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.245989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-lib-modules\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246019    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-netns\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246042    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-tmp-dir\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246050    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-bin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246064    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-sys\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.246129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246124    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-slash\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqk7b\" (UniqueName: \"kubernetes.io/projected/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-kube-api-access-sqk7b\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246158    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246211    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-conf-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246215    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-dbus\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jh6m\" (UniqueName: \"kubernetes.io/projected/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-kube-api-access-4jh6m\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246286    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-netns\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-device-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.246358 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246337    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-device-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.246635 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246357    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nz7\" (UniqueName: \"kubernetes.io/projected/9fbf954b-1744-434a-8f0f-7ac3d546e081-kube-api-access-98nz7\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.246635 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246394    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-multus-certs\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246635 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246421    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-bin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246635 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246495    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-sys\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.246635 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-kubelet-config\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.246635 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-tuned\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.246911 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ef68d50-d2d2-4244-848f-208cf21b039a-konnectivity-ca\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.246911 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246564    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.246911 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246571    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-etc-kubernetes\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.246911 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246895    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-config\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.247098 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246960    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-lib-modules\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.247098 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.246996    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:54:23.247098 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-socket-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.247098 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247083    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-var-lib-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-log-socket\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-system-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247147    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-log-socket\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247228    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-hostroot\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247257    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-socket-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247270    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-var-lib-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.247293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-socket-dir-parent\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247304    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-hostroot\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247334    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-socket-dir-parent\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-kubelet-config\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247481    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-system-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysconfig\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa50191b-3520-4499-a385-a0e6a40c3da5-host-slash\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247580    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247609    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-systemd-units\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.247668 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247639    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-bin\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247700    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-multus-daemon-config\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247725    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-cnibin\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247788    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-systemd-units\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247809    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa50191b-3520-4499-a385-a0e6a40c3da5-host-slash\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247849    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247880    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6f47\" (UniqueName: \"kubernetes.io/projected/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-kube-api-access-r6f47\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247866    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysconfig\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-registration-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247929    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-cnibin\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247944    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-env-overrides\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.247933    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247988    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bppxt\" (UniqueName: \"kubernetes.io/projected/27151663-70b9-4779-9730-fd65ee35bfcb-kube-api-access-bppxt\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.248020    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret podName:ccbc2f47-f780-45bf-8d91-e2d081bab080 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:23.747995819 +0000 UTC m=+3.140744970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret") pod "global-pull-secret-syncer-lhtq6" (UID: "ccbc2f47-f780-45bf-8d91-e2d081bab080") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.247943    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-bin\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248053    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-conf\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.248227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-host\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248135    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa50191b-3520-4499-a385-a0e6a40c3da5-iptables-alerter-script\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248163    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ef68d50-d2d2-4244-848f-208cf21b039a-agent-certs\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248190    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248193    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-host\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248372    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248594    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-conf\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248703    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ef68d50-d2d2-4244-848f-208cf21b039a-konnectivity-ca\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248736    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-registration-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-ovn\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bjg\" (UniqueName: \"kubernetes.io/projected/fe685890-13f3-4526-921b-56a13e3dcea6-kube-api-access-q8bjg\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248825    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248839    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-systemd\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-tmp\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248894    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-netns\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.248922 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-node-log\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249477 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249316    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa50191b-3520-4499-a385-a0e6a40c3da5-iptables-alerter-script\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.249477 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249319    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-multus-daemon-config\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.249477 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.248952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249477 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27151663-70b9-4779-9730-fd65ee35bfcb-ovn-node-metrics-cert\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249477 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249450    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-ovn\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249477 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249475    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-script-lib\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-cni-binary-copy\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-modprobe-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-sys-fs\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249608    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-env-overrides\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249677    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-sys-fs\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249679    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-netns\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249725    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-node-log\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249412    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-systemd\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.249795 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.249727    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.250266 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.250244    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-script-lib\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.250347 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.250292    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-cni-binary-copy\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.250427 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.250360    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-modprobe-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.251650 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.251576    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-tuned\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.252381 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.252358    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-tmp\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.254357 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.254338    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27151663-70b9-4779-9730-fd65ee35bfcb-ovn-node-metrics-cert\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.260723 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.260689    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ef68d50-d2d2-4244-848f-208cf21b039a-agent-certs\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.286555 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.286515    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcng4\" (UniqueName: \"kubernetes.io/projected/f106e28a-9a25-4d6b-a16f-add5d5591b60-kube-api-access-hcng4\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.290267 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.290240    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqvb\" (UniqueName: \"kubernetes.io/projected/453941ab-5ec2-4b68-9991-492690f822be-kube-api-access-rrqvb\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.291674 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.291648    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jh6m\" (UniqueName: \"kubernetes.io/projected/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-kube-api-access-4jh6m\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.292467 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.292443    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nz7\" (UniqueName: \"kubernetes.io/projected/9fbf954b-1744-434a-8f0f-7ac3d546e081-kube-api-access-98nz7\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.292639 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.292623    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:23.292708 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.292647    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:23.292708 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.292660    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7bgb for pod openshift-network-diagnostics/network-check-target-mrnst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:23.292799 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.292729    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb podName:09ab82b8-78fe-4884-9776-be21a025ec67 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:23.792710723 +0000 UTC m=+3.185459887 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k7bgb" (UniqueName: "kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb") pod "network-check-target-mrnst" (UID: "09ab82b8-78fe-4884-9776-be21a025ec67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:23.292799 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.292743    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdcj\" (UniqueName: \"kubernetes.io/projected/fa50191b-3520-4499-a385-a0e6a40c3da5-kube-api-access-6zdcj\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.295235 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.295216    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqk7b\" (UniqueName: \"kubernetes.io/projected/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-kube-api-access-sqk7b\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.308226 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.308181    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bppxt\" (UniqueName: \"kubernetes.io/projected/27151663-70b9-4779-9730-fd65ee35bfcb-kube-api-access-bppxt\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.308409 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.308387    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6f47\" (UniqueName: \"kubernetes.io/projected/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-kube-api-access-r6f47\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.308490 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.308463    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bjg\" (UniqueName: \"kubernetes.io/projected/fe685890-13f3-4526-921b-56a13e3dcea6-kube-api-access-q8bjg\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:23.371317 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.371287    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:54:23.375655 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.375638    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:54:23.390020 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.389995    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbf954b_1744_434a_8f0f_7ac3d546e081.slice/crio-8826e3848a5bc8c898216cb12d22dac13cb13c3ac1f1c1ae82f748420927f065 WatchSource:0}: Error finding container 8826e3848a5bc8c898216cb12d22dac13cb13c3ac1f1c1ae82f748420927f065: Status 404 returned error can't find the container with id 8826e3848a5bc8c898216cb12d22dac13cb13c3ac1f1c1ae82f748420927f065
Oct 30 02:54:23.391114 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.391098    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5bwp8"
Oct 30 02:54:23.397445 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.397426    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:23.401890 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.401867    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453941ab_5ec2_4b68_9991_492690f822be.slice/crio-c4c951371de2e965f6faf8c75159dd5935825f634d130162fa7852b952797157 WatchSource:0}: Error finding container c4c951371de2e965f6faf8c75159dd5935825f634d130162fa7852b952797157: Status 404 returned error can't find the container with id c4c951371de2e965f6faf8c75159dd5935825f634d130162fa7852b952797157
Oct 30 02:54:23.408274 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.408242    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27151663_70b9_4779_9730_fd65ee35bfcb.slice/crio-d6bc118809ce20fcb0cf69f5b11c4acae308418b94bf8b8ab7fe72e86a0d6298 WatchSource:0}: Error finding container d6bc118809ce20fcb0cf69f5b11c4acae308418b94bf8b8ab7fe72e86a0d6298: Status 404 returned error can't find the container with id d6bc118809ce20fcb0cf69f5b11c4acae308418b94bf8b8ab7fe72e86a0d6298
Oct 30 02:54:23.408833 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.408787    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:54:23.412997 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.412973    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:54:23.415477 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.415450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:54:23.418668 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.418645    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0bfb91_a5ed_41c0_a26e_fe66447eda3b.slice/crio-ae638a362e279d8f7278de13c2a81848b500ae188e5f621a5c6e6ee88c6f321f WatchSource:0}: Error finding container ae638a362e279d8f7278de13c2a81848b500ae188e5f621a5c6e6ee88c6f321f: Status 404 returned error can't find the container with id ae638a362e279d8f7278de13c2a81848b500ae188e5f621a5c6e6ee88c6f321f
Oct 30 02:54:23.420415 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.420395    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:23.427331 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.427314    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:54:23.433291 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.433148    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa50191b_3520_4499_a385_a0e6a40c3da5.slice/crio-3c23ce0aafb5479699870ce8ce861e9ba09579e16ff6a1908a7de95481e70b33 WatchSource:0}: Error finding container 3c23ce0aafb5479699870ce8ce861e9ba09579e16ff6a1908a7de95481e70b33: Status 404 returned error can't find the container with id 3c23ce0aafb5479699870ce8ce861e9ba09579e16ff6a1908a7de95481e70b33
Oct 30 02:54:23.433490 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.433474    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf106e28a_9a25_4d6b_a16f_add5d5591b60.slice/crio-736df77af67d2c9d88e61af84f6c786f8fc4da888f60ff02ba560cb4f0dfc72a WatchSource:0}: Error finding container 736df77af67d2c9d88e61af84f6c786f8fc4da888f60ff02ba560cb4f0dfc72a: Status 404 returned error can't find the container with id 736df77af67d2c9d88e61af84f6c786f8fc4da888f60ff02ba560cb4f0dfc72a
Oct 30 02:54:23.434038 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.434019    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:54:23.439990 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.439846    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef68d50_d2d2_4244_848f_208cf21b039a.slice/crio-badf764c0fead9bee1750f6db27a4bac70c86f9d70cab2fa9e9eeb953752f6a2 WatchSource:0}: Error finding container badf764c0fead9bee1750f6db27a4bac70c86f9d70cab2fa9e9eeb953752f6a2: Status 404 returned error can't find the container with id badf764c0fead9bee1750f6db27a4bac70c86f9d70cab2fa9e9eeb953752f6a2
Oct 30 02:54:23.443310 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.443274    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b96a13_ca71_48c9_9fa1_6d785e9e3b3c.slice/crio-a496efaf565d8e0d4f42c5c07863f46efa3165398d5336eb6a6ecaba850f3a9a WatchSource:0}: Error finding container a496efaf565d8e0d4f42c5c07863f46efa3165398d5336eb6a6ecaba850f3a9a: Status 404 returned error can't find the container with id a496efaf565d8e0d4f42c5c07863f46efa3165398d5336eb6a6ecaba850f3a9a
Oct 30 02:54:23.445662 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:23.445640    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd0bf35_fef8_4975_90f7_7bf6c1b70d7f.slice/crio-bd71a155762ae372acae0b06290c7e49a6b42beb7f6676287bb017fe649adbc5 WatchSource:0}: Error finding container bd71a155762ae372acae0b06290c7e49a6b42beb7f6676287bb017fe649adbc5: Status 404 returned error can't find the container with id bd71a155762ae372acae0b06290c7e49a6b42beb7f6676287bb017fe649adbc5
Oct 30 02:54:23.753314 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.753116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:23.753485 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.753359    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:23.753485 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.753364    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:23.753485 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.753434    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret podName:ccbc2f47-f780-45bf-8d91-e2d081bab080 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:24.75341569 +0000 UTC m=+4.146164854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret") pod "global-pull-secret-syncer-lhtq6" (UID: "ccbc2f47-f780-45bf-8d91-e2d081bab080") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:23.753485 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.753451    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:23.753715 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.753505    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:24.753488421 +0000 UTC m=+4.146237583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:23.854847 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:23.854192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:23.854847 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.854413    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:23.854847 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.854437    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:23.854847 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.854450    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7bgb for pod openshift-network-diagnostics/network-check-target-mrnst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:23.854847 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:23.854506    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb podName:09ab82b8-78fe-4884-9776-be21a025ec67 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:24.854489399 +0000 UTC m=+4.247238546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7bgb" (UniqueName: "kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb") pod "network-check-target-mrnst" (UID: "09ab82b8-78fe-4884-9776-be21a025ec67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:24.101550 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.101398    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:23 +0000 UTC" deadline="2028-07-26 02:39:18.379324926 +0000 UTC"
Oct 30 02:54:24.101550 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.101456    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15239h44m54.27787306s"
Oct 30 02:54:24.164063 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.164034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:24.164222 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.164169    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:24.186972 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.186135    2574 generic.go:356] "Generic (PLEG): container finished" podID="77d685204da0cc5fa69ed201b4cfe0a4" containerID="71bb6fc09d82a11d5c1e06bccc66b6912240e63ac0a27ea0d4217bd56ecbe15d" exitCode=0
Oct 30 02:54:24.186972 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.186752    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerDied","Data":"71bb6fc09d82a11d5c1e06bccc66b6912240e63ac0a27ea0d4217bd56ecbe15d"}
Oct 30 02:54:24.192404 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.192132    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x8h8g" event={"ID":"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f","Type":"ContainerStarted","Data":"bd71a155762ae372acae0b06290c7e49a6b42beb7f6676287bb017fe649adbc5"}
Oct 30 02:54:24.206481 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.206438    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zq2m7" event={"ID":"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c","Type":"ContainerStarted","Data":"a496efaf565d8e0d4f42c5c07863f46efa3165398d5336eb6a6ecaba850f3a9a"}
Oct 30 02:54:24.223382 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.223279    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j9s8d" event={"ID":"3ef68d50-d2d2-4244-848f-208cf21b039a","Type":"ContainerStarted","Data":"badf764c0fead9bee1750f6db27a4bac70c86f9d70cab2fa9e9eeb953752f6a2"}
Oct 30 02:54:24.238343 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.238246    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fttld" event={"ID":"fa50191b-3520-4499-a385-a0e6a40c3da5","Type":"ContainerStarted","Data":"3c23ce0aafb5479699870ce8ce861e9ba09579e16ff6a1908a7de95481e70b33"}
Oct 30 02:54:24.241609 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.241547    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerStarted","Data":"736df77af67d2c9d88e61af84f6c786f8fc4da888f60ff02ba560cb4f0dfc72a"}
Oct 30 02:54:24.250946 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.250899    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"d6bc118809ce20fcb0cf69f5b11c4acae308418b94bf8b8ab7fe72e86a0d6298"}
Oct 30 02:54:24.255677 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.255407    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"d1b766764a60d5387ea6838359f9932270b971a5fcf845f89d0a4e6b28b701de"}
Oct 30 02:54:24.259322 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.259296    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mwh8j" event={"ID":"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b","Type":"ContainerStarted","Data":"ae638a362e279d8f7278de13c2a81848b500ae188e5f621a5c6e6ee88c6f321f"}
Oct 30 02:54:24.262342 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.262312    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5bwp8" event={"ID":"453941ab-5ec2-4b68-9991-492690f822be","Type":"ContainerStarted","Data":"c4c951371de2e965f6faf8c75159dd5935825f634d130162fa7852b952797157"}
Oct 30 02:54:24.272902 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.272839    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f" event={"ID":"9fbf954b-1744-434a-8f0f-7ac3d546e081","Type":"ContainerStarted","Data":"8826e3848a5bc8c898216cb12d22dac13cb13c3ac1f1c1ae82f748420927f065"}
Oct 30 02:54:24.300448 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.299318    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" podStartSLOduration=2.299302489 podStartE2EDuration="2.299302489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:24.299201878 +0000 UTC m=+3.691951045" watchObservedRunningTime="2026-10-30 02:54:24.299302489 +0000 UTC m=+3.692051657"
Oct 30 02:54:24.761451 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.761363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:24.761451 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.761427    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:24.761688 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.761563    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:24.761688 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.761622    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:26.761604612 +0000 UTC m=+6.154353771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:24.761999 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.761982    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:24.762067 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.762039    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret podName:ccbc2f47-f780-45bf-8d91-e2d081bab080 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:26.762022153 +0000 UTC m=+6.154771312 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret") pod "global-pull-secret-syncer-lhtq6" (UID: "ccbc2f47-f780-45bf-8d91-e2d081bab080") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:24.863400 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:24.863363    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:24.863592 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.863513    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:24.863592 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.863549    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:24.863592 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.863563    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7bgb for pod openshift-network-diagnostics/network-check-target-mrnst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:24.863750 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:24.863617    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb podName:09ab82b8-78fe-4884-9776-be21a025ec67 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:26.863599829 +0000 UTC m=+6.256348988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7bgb" (UniqueName: "kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb") pod "network-check-target-mrnst" (UID: "09ab82b8-78fe-4884-9776-be21a025ec67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:25.102165 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:25.102075    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:23 +0000 UTC" deadline="2028-08-05 21:09:18.477079734 +0000 UTC"
Oct 30 02:54:25.102165 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:25.102111    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15498h14m53.3749727s"
Oct 30 02:54:25.166892 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:25.166861    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:25.167087 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:25.166998    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:25.167452 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:25.167427    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:25.167561 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:25.167546    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:25.285203 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:25.285168    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"e36441d99636c9836e8e5e086d29aa0768b1c7d2ed873ee09b1d8780425c5c4f"}
Oct 30 02:54:26.163940 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:26.163867    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:26.164409 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.163982    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:26.289840 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:26.289803    2574 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="1d212cbbafcbd087ec1f3e7fe77a431c6a30e5267456219ebcc14a68198690ef" exitCode=0
Oct 30 02:54:26.290013 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:26.289929    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerDied","Data":"1d212cbbafcbd087ec1f3e7fe77a431c6a30e5267456219ebcc14a68198690ef"}
Oct 30 02:54:26.371927 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:26.371386    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" podStartSLOduration=4.371365698 podStartE2EDuration="4.371365698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:25.329578225 +0000 UTC m=+4.722327390" watchObservedRunningTime="2026-10-30 02:54:26.371365698 +0000 UTC m=+5.764114866"
Oct 30 02:54:26.779775 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:26.779736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:26.779965 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:26.779819    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:26.779965 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.779937    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:26.780077 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.779994    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret podName:ccbc2f47-f780-45bf-8d91-e2d081bab080 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:30.779978097 +0000 UTC m=+10.172727254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret") pod "global-pull-secret-syncer-lhtq6" (UID: "ccbc2f47-f780-45bf-8d91-e2d081bab080") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:26.780407 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.780386    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:26.780505 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.780442    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:30.780426429 +0000 UTC m=+10.173175587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:26.881688 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:26.881047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:26.881688 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.881245    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:26.881688 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.881264    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:26.881688 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.881276    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7bgb for pod openshift-network-diagnostics/network-check-target-mrnst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:26.881688 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:26.881332    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb podName:09ab82b8-78fe-4884-9776-be21a025ec67 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:30.881314211 +0000 UTC m=+10.274063361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7bgb" (UniqueName: "kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb") pod "network-check-target-mrnst" (UID: "09ab82b8-78fe-4884-9776-be21a025ec67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:27.166065 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:27.165356    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:27.166065 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:27.165510    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:27.166065 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:27.165900    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:27.166065 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:27.165988    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:28.163293 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:28.163254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:28.163492 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:28.163396    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:28.298803 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:28.298762    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-j9s8d" event={"ID":"3ef68d50-d2d2-4244-848f-208cf21b039a","Type":"ContainerStarted","Data":"af16c4cb48be20aeb78836d72f49f304d322691099caadb9ac9dd9f97a1abbf5"}
Oct 30 02:54:29.166514 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.166441    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:29.166690 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:29.166612    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:29.167144 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.167001    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:29.167144 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:29.167084    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:29.304945 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.304909    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zq2m7" event={"ID":"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c","Type":"ContainerStarted","Data":"c1fa86ace9b4611a9448717f310905f08bc53caa6c1ffe284c41e0a89ddb3929"}
Oct 30 02:54:29.308181 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.308153    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f" event={"ID":"9fbf954b-1744-434a-8f0f-7ac3d546e081","Type":"ContainerStarted","Data":"53fa56b185206a89c642d05b4f749b012674a0926b4a5db7cc480cccd53894c2"}
Oct 30 02:54:29.329112 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.329075    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:29.329782 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.329727    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:29.349868 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.349815    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-j9s8d" podStartSLOduration=4.666388476 podStartE2EDuration="8.349797957s" totalImagesPullingTime="3.683409481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:28.349833229 +0000 UTC m=+7.742582395" watchObservedRunningTime="2026-10-30 02:54:29.349797957 +0000 UTC m=+8.742547125"
Oct 30 02:54:29.350048 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:29.349920    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zq2m7" podStartSLOduration=3.452842037 podStartE2EDuration="8.349914011s" totalImagesPullingTime="4.897071974s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:29.349587953 +0000 UTC m=+8.742337121" watchObservedRunningTime="2026-10-30 02:54:29.349914011 +0000 UTC m=+8.742663178"
Oct 30 02:54:30.163142 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:30.163114    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:30.163332 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.163234    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:30.354810 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:30.354778    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:54:30.815123 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:30.815086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:30.815257 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:30.815158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:30.815317 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.815308    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:30.815389 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.815374    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:38.815355926 +0000 UTC m=+18.208105084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:30.816551 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.815802    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:30.816551 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.815860    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret podName:ccbc2f47-f780-45bf-8d91-e2d081bab080 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:38.815844212 +0000 UTC m=+18.208593370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret") pod "global-pull-secret-syncer-lhtq6" (UID: "ccbc2f47-f780-45bf-8d91-e2d081bab080") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:30.916575 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:30.916270    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:30.916575 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.916420    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:30.916575 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.916439    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:30.916575 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.916452    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7bgb for pod openshift-network-diagnostics/network-check-target-mrnst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:30.916575 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:30.916506    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb podName:09ab82b8-78fe-4884-9776-be21a025ec67 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:38.916489066 +0000 UTC m=+18.309238228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7bgb" (UniqueName: "kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb") pod "network-check-target-mrnst" (UID: "09ab82b8-78fe-4884-9776-be21a025ec67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:31.096485 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.096367    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:54:30.354805853Z","UUID":"0724d6e7-e8f1-41e0-b05a-157444712371","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:54:31.098057 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.098035    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:54:31.098195 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.098066    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:54:31.165129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.164541    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:31.165129 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:31.164646    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:31.165129 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.165004    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:31.165129 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:31.165099    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:31.316143 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.315901    2574 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="545de7aae91fe292b18fed27e51929ec3c13e2f75d381a0ec4423acb24a8c8de" exitCode=0
Oct 30 02:54:31.316227 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.316094    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerDied","Data":"545de7aae91fe292b18fed27e51929ec3c13e2f75d381a0ec4423acb24a8c8de"}
Oct 30 02:54:31.320670 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.320636    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5bwp8" event={"ID":"453941ab-5ec2-4b68-9991-492690f822be","Type":"ContainerStarted","Data":"5a5767ee68d4dd98a337835a96e449e5b213b3201522ea3693caa5e14be5a3a4"}
Oct 30 02:54:31.324371 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.324348    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f" event={"ID":"9fbf954b-1744-434a-8f0f-7ac3d546e081","Type":"ContainerStarted","Data":"55d72d74660509582b036282d564ad44dd4236be21721900d0a124d222c6334b"}
Oct 30 02:54:31.327492 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.327471    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:54:31.327603 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.327511    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-x8h8g" event={"ID":"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f","Type":"ContainerStarted","Data":"5d94449e4397958aa6d15199197ef428f28b593773b30fe396655b29c7cffac8"}
Oct 30 02:54:31.419632 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.419422    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-5bwp8" podStartSLOduration=3.046263305 podStartE2EDuration="10.419406148s" totalImagesPullingTime="7.373142843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:31.418654177 +0000 UTC m=+10.811403344" watchObservedRunningTime="2026-10-30 02:54:31.419406148 +0000 UTC m=+10.812155314"
Oct 30 02:54:31.462956 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.462905    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-x8h8g" podStartSLOduration=2.773311108 podStartE2EDuration="10.462892068s" totalImagesPullingTime="7.68958096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:31.462129878 +0000 UTC m=+10.854879063" watchObservedRunningTime="2026-10-30 02:54:31.462892068 +0000 UTC m=+10.855641233"
Oct 30 02:54:31.550853 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.550820    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:31.551806 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:31.551653    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:54:32.163229 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:32.163153    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:32.163391 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:32.163290    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:32.332475 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:32.332433    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f" event={"ID":"9fbf954b-1744-434a-8f0f-7ac3d546e081","Type":"ContainerStarted","Data":"ada1965a0076a94e659cf58aa7e1b44eb883bbe225946fd925fbf6aec7910feb"}
Oct 30 02:54:32.335084 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:32.335055    2574 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="cdc3eed5bf0faae97dd0743d3968ea5ad85e37a520b72b40237226272f9fe338" exitCode=0
Oct 30 02:54:32.335279 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:32.335152    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerDied","Data":"cdc3eed5bf0faae97dd0743d3968ea5ad85e37a520b72b40237226272f9fe338"}
Oct 30 02:54:32.382102 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:32.382042    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f" podStartSLOduration=3.70613532 podStartE2EDuration="11.382023765s" totalImagesPullingTime="7.675888445s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:32.381973139 +0000 UTC m=+11.774722286" watchObservedRunningTime="2026-10-30 02:54:32.382023765 +0000 UTC m=+11.774772933"
Oct 30 02:54:33.163864 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:33.163820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:33.164357 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:33.163965    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:33.164357 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:33.164267    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:33.164482 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:33.164374    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:33.338964 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:33.338871    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fttld" event={"ID":"fa50191b-3520-4499-a385-a0e6a40c3da5","Type":"ContainerStarted","Data":"df7a19275c1177b8c32d2fcc48e87154391a71ddc1b54aa376fffc4b61bfc849"}
Oct 30 02:54:33.383763 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:33.383712    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fttld" podStartSLOduration=4.507302695 podStartE2EDuration="12.383696954s" totalImagesPullingTime="7.876394259s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:33.383274215 +0000 UTC m=+12.776023383" watchObservedRunningTime="2026-10-30 02:54:33.383696954 +0000 UTC m=+12.776446119"
Oct 30 02:54:34.163912 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:34.163880    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:34.164495 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:34.164001    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:34.342384 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:34.342342    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mwh8j" event={"ID":"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b","Type":"ContainerStarted","Data":"a8ea69c852a63ccd8e249ecc60deacfdde42b6691b7141e49ec74d4ae29b5b8b"}
Oct 30 02:54:34.345230 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:34.345198    2574 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="6e7143e9417f6ed5663b45d66fc23e0edb58849f919f10e1597563a610f11628" exitCode=0
Oct 30 02:54:34.345359 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:34.345275    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerDied","Data":"6e7143e9417f6ed5663b45d66fc23e0edb58849f919f10e1597563a610f11628"}
Oct 30 02:54:34.389221 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:34.389161    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mwh8j" podStartSLOduration=3.334762869 podStartE2EDuration="13.389146828s" totalImagesPullingTime="10.054383959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:34.388725501 +0000 UTC m=+13.781474676" watchObservedRunningTime="2026-10-30 02:54:34.389146828 +0000 UTC m=+13.781895993"
Oct 30 02:54:35.164152 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:35.164120    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:35.164773 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:35.164250    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:35.164773 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:35.164698    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:35.164891 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:35.164816    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:36.164173 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:36.164005    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:36.164556 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:36.164259    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:36.352983 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:36.352948    2574 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="0bd4092514a6e49bde34138aa29f2ee7ff0f7fbb2bce8b896947b2e772a3da99" exitCode=0
Oct 30 02:54:36.353130 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:36.353010    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerDied","Data":"0bd4092514a6e49bde34138aa29f2ee7ff0f7fbb2bce8b896947b2e772a3da99"}
Oct 30 02:54:37.163141 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:37.163108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:37.163379 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:37.163108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:37.163379 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:37.163250    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:37.163379 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:37.163285    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:37.358424 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:37.358390    2574 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="d75ca9a673234451598fe81cd1f7fa75f82d84703c4370f58a34589f12bf1e67" exitCode=0
Oct 30 02:54:37.358797 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:37.358439    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerDied","Data":"d75ca9a673234451598fe81cd1f7fa75f82d84703c4370f58a34589f12bf1e67"}
Oct 30 02:54:38.164029 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.163852    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:38.164196 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.164096    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:38.363769 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.363682    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6x562" event={"ID":"f106e28a-9a25-4d6b-a16f-add5d5591b60","Type":"ContainerStarted","Data":"2ace7ecfaa6ff0dca3476fa8aa719f7c3668b3b4263e97aa9c9efe5fab3b7abc"}
Oct 30 02:54:38.366827 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.366805    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"892ac225e6a46a4aff0eb12ac89142aae044266e42f5aacf21d6f39954f702a6"}
Oct 30 02:54:38.366936 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.366857    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"50a1a8c293aee0868630afc03e26f5c1b66559d0db77c18d6cae3a70695e232a"}
Oct 30 02:54:38.366936 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.366885    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"97bf5ec9e0a3b7cc8afd6b038d06f092b27f46bb98f5faceed1febe6abca130f"}
Oct 30 02:54:38.366936 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.366913    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"d0043000b853a1014ad60459d1abca52880e13f1831b635dcb7340883bc86da0"}
Oct 30 02:54:38.367030 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.366940    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"d0465fdf913e11f0324549f691dae127e17c8eac9b76830b82a72bea7bcd1a95"}
Oct 30 02:54:38.367030 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.366966    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"065e66f6a9e3869fbd430335b25338a503517c8156127f03516d013f9e76c355"}
Oct 30 02:54:38.414994 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.414952    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6x562" podStartSLOduration=7.070988826 podStartE2EDuration="17.414940841s" totalImagesPullingTime="10.343952015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:38.414416601 +0000 UTC m=+17.807165766" watchObservedRunningTime="2026-10-30 02:54:38.414940841 +0000 UTC m=+17.807690006"
Oct 30 02:54:38.872915 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.872885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:38.873121 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.872941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:38.873121 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.873034    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:38.873121 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.873035    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:38.873121 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.873090    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.873075515 +0000 UTC m=+34.265824660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 02:54:38.873121 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.873124    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret podName:ccbc2f47-f780-45bf-8d91-e2d081bab080 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.87311456 +0000 UTC m=+34.265863705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret") pod "global-pull-secret-syncer-lhtq6" (UID: "ccbc2f47-f780-45bf-8d91-e2d081bab080") : object "kube-system"/"original-pull-secret" not registered
Oct 30 02:54:38.973497 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:38.973460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:38.973702 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.973673    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 02:54:38.973702 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.973696    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 02:54:38.973803 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.973710    2574 projected.go:196] Error preparing data for projected volume kube-api-access-k7bgb for pod openshift-network-diagnostics/network-check-target-mrnst: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:38.973803 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:38.973781    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb podName:09ab82b8-78fe-4884-9776-be21a025ec67 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.973761968 +0000 UTC m=+34.366511130 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k7bgb" (UniqueName: "kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb") pod "network-check-target-mrnst" (UID: "09ab82b8-78fe-4884-9776-be21a025ec67") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 02:54:39.163959 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:39.163864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:39.163959 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:39.163881    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:39.164156 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:39.163974    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:39.164156 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:39.164111    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:40.163326 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:40.163281    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:40.163867 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:40.163381    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:40.376095 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:40.376015    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"7f82dd10437bd6885844100d3ffbefb02a44de450de94403a91756ad34002f09"}
Oct 30 02:54:41.164363 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:41.164324    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:41.164903 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:41.164464    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:41.164903 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:41.164499    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:41.164903 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:41.164570    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:42.163296 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.163276    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:42.163419 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:42.163396    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:42.383494 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.383458    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" event={"ID":"27151663-70b9-4779-9730-fd65ee35bfcb","Type":"ContainerStarted","Data":"6f14e524585f6daee3de6af04201d5c2ced84f8744d57727034fc59657c0a22d"}
Oct 30 02:54:42.384099 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.383659    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:42.384099 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.383717    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:42.384099 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.383767    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:42.398655 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.398628    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:42.398774 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.398732    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:54:42.483108 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:42.482995    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt" podStartSLOduration=7.152819206 podStartE2EDuration="21.482983135s" totalImagesPullingTime="14.330163929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:42.482461725 +0000 UTC m=+21.875210890" watchObservedRunningTime="2026-10-30 02:54:42.482983135 +0000 UTC m=+21.875732301"
Oct 30 02:54:43.163778 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:43.163748    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:43.164035 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:43.163859    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:43.164035 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:43.163916    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:43.164035 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:43.164011    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:44.164006 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:44.163977    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:44.164441 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:44.164112    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:44.257699 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:44.257626    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhtq6"]
Oct 30 02:54:44.258215 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:44.258195    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s258j"]
Oct 30 02:54:44.258332 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:44.258321    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:44.258429 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:44.258412    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:44.271787 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:44.271758    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrnst"]
Oct 30 02:54:44.271885 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:44.271864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:44.271958 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:44.271941    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:44.388274 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:44.388244    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:44.388456 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:44.388369    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:46.163372 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:46.163341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:46.163907 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:46.163378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:46.163907 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:46.163383    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:46.163907 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:46.163457    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:46.163907 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:46.163562    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:46.163907 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:46.163652    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:48.163130 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:48.163093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:48.163130 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:48.163093    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:48.163625 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:48.163110    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:48.163625 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:48.163209    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:48.163625 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:48.163291    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:48.163625 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:48.163360    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:50.163537 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.163485    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:50.164020 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.163547    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:50.164020 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.163499    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:50.164020 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.163626    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mrnst" podUID="09ab82b8-78fe-4884-9776-be21a025ec67"
Oct 30 02:54:50.164020 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.163696    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s258j" podUID="fe685890-13f3-4526-921b-56a13e3dcea6"
Oct 30 02:54:50.164020 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.163745    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lhtq6" podUID="ccbc2f47-f780-45bf-8d91-e2d081bab080"
Oct 30 02:54:50.396311 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.396288    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeReady"
Oct 30 02:54:50.396451 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.396404    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 02:54:50.461491 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.461422    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-h9c28"]
Oct 30 02:54:50.467094 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.467071    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jtjnj"]
Oct 30 02:54:50.467240 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.467216    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.470175 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.470150    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.484607 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.484589    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvbgx\""
Oct 30 02:54:50.485781 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.485763    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9c28"]
Oct 30 02:54:50.487904 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.487866    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:54:50.488968 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.488946    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jtjnj"]
Oct 30 02:54:50.498889 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.498861    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:54:50.498889 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.498860    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:54:50.499439 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.499422    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rq99w"]
Oct 30 02:54:50.502101 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.502087    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:50.523660 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.523639    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rq99w"]
Oct 30 02:54:50.555180 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555152    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwkm\" (UniqueName: \"kubernetes.io/projected/027c260d-eae4-4dda-834e-be9433d32534-kube-api-access-hxwkm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.555282 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555192    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252025-152e-4341-9a75-285b38cc2436-tmp-dir\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.555282 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t27bz\" (UniqueName: \"kubernetes.io/projected/e2252025-152e-4341-9a75-285b38cc2436-kube-api-access-t27bz\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.555352 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555328    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/027c260d-eae4-4dda-834e-be9433d32534-data-volume\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.555384 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.555384 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.555444 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555395    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252025-152e-4341-9a75-285b38cc2436-config-volume\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.555478 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.555506 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.555479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/027c260d-eae4-4dda-834e-be9433d32534-crio-socket\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.587435 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.587406    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:54:50.587435 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.587410    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:54:50.587585 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.587540    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:54:50.587701 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.587540    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkhcx\""
Oct 30 02:54:50.587996 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.587964    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4tlgk\""
Oct 30 02:54:50.588225 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.588202    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:54:50.588320 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.588302    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:54:50.588465 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.588446    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:54:50.656313 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656283    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4663c2-fffd-489d-8f4b-aa26181bff45-cert\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:50.656464 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656323    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.656464 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656343    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252025-152e-4341-9a75-285b38cc2436-config-volume\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.656464 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656367    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.656464 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.656424    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.656471    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.656491    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:51.156471089 +0000 UTC m=+30.549220251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/027c260d-eae4-4dda-834e-be9433d32534-crio-socket\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.656554    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:51.156511585 +0000 UTC m=+30.549260743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwkm\" (UniqueName: \"kubernetes.io/projected/027c260d-eae4-4dda-834e-be9433d32534-kube-api-access-hxwkm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252025-152e-4341-9a75-285b38cc2436-tmp-dir\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656691    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/027c260d-eae4-4dda-834e-be9433d32534-crio-socket\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.656707 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656699    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27bz\" (UniqueName: \"kubernetes.io/projected/e2252025-152e-4341-9a75-285b38cc2436-kube-api-access-t27bz\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.657149 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s9cq\" (UniqueName: \"kubernetes.io/projected/6c4663c2-fffd-489d-8f4b-aa26181bff45-kube-api-access-7s9cq\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:50.657149 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/027c260d-eae4-4dda-834e-be9433d32534-data-volume\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.657149 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.656853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.657149 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.656967    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:50.657149 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:50.657009    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:51.156995557 +0000 UTC m=+30.549744716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:50.657149 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.657103    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252025-152e-4341-9a75-285b38cc2436-tmp-dir\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.657149 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.657126    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252025-152e-4341-9a75-285b38cc2436-config-volume\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.657381 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.657163    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/027c260d-eae4-4dda-834e-be9433d32534-data-volume\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.684255 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.684139    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwkm\" (UniqueName: \"kubernetes.io/projected/027c260d-eae4-4dda-834e-be9433d32534-kube-api-access-hxwkm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:50.684353 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.684140    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27bz\" (UniqueName: \"kubernetes.io/projected/e2252025-152e-4341-9a75-285b38cc2436-kube-api-access-t27bz\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:50.757703 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.757619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9cq\" (UniqueName: \"kubernetes.io/projected/6c4663c2-fffd-489d-8f4b-aa26181bff45-kube-api-access-7s9cq\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:50.757808 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.757769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4663c2-fffd-489d-8f4b-aa26181bff45-cert\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:50.760020 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.760001    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4663c2-fffd-489d-8f4b-aa26181bff45-cert\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:50.782852 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.782822    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9cq\" (UniqueName: \"kubernetes.io/projected/6c4663c2-fffd-489d-8f4b-aa26181bff45-kube-api-access-7s9cq\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:50.811738 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:50.811717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:54:51.160743 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:51.160721    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rq99w"]
Oct 30 02:54:51.161008 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:51.160991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:51.161085 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:51.161029    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:51.161085 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:51.161060    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:51.161293 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:51.161131    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:51.161293 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:51.161156    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:51.161293 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:51.161130    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:51.161293 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:51.161192    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:52.161173311 +0000 UTC m=+31.553922472 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:51.161293 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:51.161239    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:52.161222757 +0000 UTC m=+31.553971912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:51.161293 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:51.161250    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:52.16124452 +0000 UTC m=+31.553993664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:51.163150 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:51.163123    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4663c2_fffd_489d_8f4b_aa26181bff45.slice/crio-3df1ed6f31b6cdd5eac119537ee44ac3408de59e35818b3dcaef1217bfe85b0d WatchSource:0}: Error finding container 3df1ed6f31b6cdd5eac119537ee44ac3408de59e35818b3dcaef1217bfe85b0d: Status 404 returned error can't find the container with id 3df1ed6f31b6cdd5eac119537ee44ac3408de59e35818b3dcaef1217bfe85b0d
Oct 30 02:54:51.404616 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:51.404580    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rq99w" event={"ID":"6c4663c2-fffd-489d-8f4b-aa26181bff45","Type":"ContainerStarted","Data":"3df1ed6f31b6cdd5eac119537ee44ac3408de59e35818b3dcaef1217bfe85b0d"}
Oct 30 02:54:52.163324 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.163289    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:52.163493 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.163382    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:52.163493 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.163477    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:52.167510 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.167485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:52.167663 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.167547    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:52.167663 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.167583    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:52.167663 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:52.167648    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:52.167813 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:52.167687    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:52.167813 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:52.167697    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:52.167813 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:52.167717    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.167697608 +0000 UTC m=+33.560446765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:52.167813 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:52.167737    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.167725952 +0000 UTC m=+33.560475116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:52.167813 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:52.167752    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:54.1677457 +0000 UTC m=+33.560494843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:52.188773 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.188751    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h9jxb\""
Oct 30 02:54:52.188879 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.188785    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l729x\""
Oct 30 02:54:52.190263 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.190244    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:54:52.190353 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.190317    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:54:52.190353 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.190332    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:54:52.190460 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.190362    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d6p9p\""
Oct 30 02:54:52.191952 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:52.191938    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:54:54.185066 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.184977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:54.185066 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.185026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:54.185066 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.185047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:54.185546 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.185117    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 02:54:54.185546 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.185156    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:54.185546 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.185156    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:54.185546 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.185168    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.185155675 +0000 UTC m=+37.577904822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : configmap "kube-rbac-proxy" not found
Oct 30 02:54:54.185546 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.185250    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.185233629 +0000 UTC m=+37.577982773 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:54.185546 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.185270    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:54:58.185262003 +0000 UTC m=+37.578011148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:54.413467 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.413433    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rq99w" event={"ID":"6c4663c2-fffd-489d-8f4b-aa26181bff45","Type":"ContainerStarted","Data":"f6217e88a5af22cf7fb1a7c2833388f0fdd401e0cd6537c45c9a8e27d7bf0102"}
Oct 30 02:54:54.459949 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.459862    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rq99w" podStartSLOduration=1.816290859 podStartE2EDuration="4.459850195s" totalImagesPullingTime="2.643559336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:54:54.459187453 +0000 UTC m=+33.851936619" watchObservedRunningTime="2026-10-30 02:54:54.459850195 +0000 UTC m=+33.852599360"
Oct 30 02:54:54.889033 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.888989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:54:54.889225 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.889100    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:54.889225 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.889180    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:54:54.889305 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:54.889270    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:26.889252865 +0000 UTC m=+66.282002012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : secret "metrics-daemon-secret" not found
Oct 30 02:54:54.891544 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.891507    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:54.910402 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.910377    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:54:54.989833 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.989804    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:54.992356 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:54.992332    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:55.185259 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:55.185176    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:54:55.256474 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:55.256209    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lhtq6"]
Oct 30 02:54:55.261980 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:55.261952    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbc2f47_f780_45bf_8d91_e2d081bab080.slice/crio-69d6b5f73f3955b2a76bfbdec7a1c110de46f4f30adf930a3a517b88e3aea820 WatchSource:0}: Error finding container 69d6b5f73f3955b2a76bfbdec7a1c110de46f4f30adf930a3a517b88e3aea820: Status 404 returned error can't find the container with id 69d6b5f73f3955b2a76bfbdec7a1c110de46f4f30adf930a3a517b88e3aea820
Oct 30 02:54:55.417324 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:55.417286    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lhtq6" event={"ID":"ccbc2f47-f780-45bf-8d91-e2d081bab080","Type":"ContainerStarted","Data":"69d6b5f73f3955b2a76bfbdec7a1c110de46f4f30adf930a3a517b88e3aea820"}
Oct 30 02:54:55.526832 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:55.526812    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mrnst"]
Oct 30 02:54:55.528394 ip-10-0-138-200 kubenswrapper[2574]: W1030 02:54:55.528370    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ab82b8_78fe_4884_9776_be21a025ec67.slice/crio-67573de29f7ffc278c446da4ea50d42488ff42b722e64eddcfff3ab4fbad3b72 WatchSource:0}: Error finding container 67573de29f7ffc278c446da4ea50d42488ff42b722e64eddcfff3ab4fbad3b72: Status 404 returned error can't find the container with id 67573de29f7ffc278c446da4ea50d42488ff42b722e64eddcfff3ab4fbad3b72
Oct 30 02:54:55.977917 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:55.977879    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x8h8g_0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f/dns-node-resolver/0.log"
Oct 30 02:54:56.421192 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:56.421153    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrnst" event={"ID":"09ab82b8-78fe-4884-9776-be21a025ec67","Type":"ContainerStarted","Data":"67573de29f7ffc278c446da4ea50d42488ff42b722e64eddcfff3ab4fbad3b72"}
Oct 30 02:54:56.947699 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:56.947665    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zq2m7_44b96a13-ca71-48c9-9fa1-6d785e9e3b3c/node-ca/0.log"
Oct 30 02:54:57.753781 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:57.753749    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rq99w_6c4663c2-fffd-489d-8f4b-aa26181bff45/serve-healthcheck-canary/0.log"
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:58.214308    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:58.214375    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:58.214401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:58.214482    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:58.214557    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:58.214579    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:06.214558146 +0000 UTC m=+45.607307306 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: E1030 02:54:58.214621    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:06.214602616 +0000 UTC m=+45.607351783 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:54:58.215057 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:58.215012    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:54:58.615634 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:58.615596    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9s8d_3ef68d50-d2d2-4244-848f-208cf21b039a/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:58.778541 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:58.778494    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9s8d_3ef68d50-d2d2-4244-848f-208cf21b039a/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:58.943319 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:58.943293    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9s8d_3ef68d50-d2d2-4244-848f-208cf21b039a/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:59.167553 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:59.167455    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9s8d_3ef68d50-d2d2-4244-848f-208cf21b039a/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:59.349724 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:59.349680    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-j9s8d_3ef68d50-d2d2-4244-848f-208cf21b039a/konnectivity-agent/0.log" limit=1024
Oct 30 02:54:59.430706 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:59.430629    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mrnst" event={"ID":"09ab82b8-78fe-4884-9776-be21a025ec67","Type":"ContainerStarted","Data":"7160b5fb7b5f46d70fc5d8891694619003e4be90529c1dda2948b1871b5b7bbb"}
Oct 30 02:54:59.430876 ip-10-0-138-200 kubenswrapper[2574]: I1030 02:54:59.430857    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:55:00.344803 ip-10-0-138-200 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 02:55:00.360903 ip-10-0-138-200 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 02:55:00.361077 ip-10-0-138-200 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 02:55:00.361129 ip-10-0-138-200 systemd[1]: kubelet.service: Consumed 2.231s CPU time.
Oct 30 02:55:00.370097 ip-10-0-138-200 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 02:55:00.437035 ip-10-0-138-200 kubenswrapper[4377]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:00.437035 ip-10-0-138-200 kubenswrapper[4377]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 02:55:00.437035 ip-10-0-138-200 kubenswrapper[4377]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:00.437035 ip-10-0-138-200 kubenswrapper[4377]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 02:55:00.439093 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439013    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.439093 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439093    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439097    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439101    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439103    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439106    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439109    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439112    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439114    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439117    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439120    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439130    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439133    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439136    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439138    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439141    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439144    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439146    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439149    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439152    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439155    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.439158 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439158    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439161    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439164    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439167    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439169    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439172    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439175    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439178    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439180    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439183    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439186    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439188    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439191    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439193    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439196    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439198    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439201    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439203    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439206    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.439672 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439209    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439211    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439214    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439216    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439219    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439222    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439224    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439227    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439229    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439232    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439235    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439237    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439240    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439242    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439245    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439249    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439253    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439257    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439260    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.440136 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439264    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439266    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439269    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439271    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439274    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439277    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439279    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439282    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439284    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439287    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439289    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439292    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439295    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439297    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439300    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439303    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439306    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439308    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439311    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439314    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.440616 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439317    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439319    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439322    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439325    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439327    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439330    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439333    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439335    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439338    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439341    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439344    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439347    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439349    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439352    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439355    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439359    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439364    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439367    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439370    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.441106 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439373    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439376    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439378    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439381    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439384    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439386    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439389    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.439391    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440187    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440195    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440198    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440201    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440204    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440207    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440210    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440214    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440218    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440221    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440224    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440226    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.441580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440229    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440232    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440234    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440237    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440239    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440243    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440246    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440249    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440251    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440254    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440257    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440259    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440262    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440264    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440267    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440270    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440272    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440275    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440278    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.442068 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440280    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440283    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440285    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440288    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440291    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440293    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440296    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440299    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440303    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440306    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440309    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440311    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440314    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440317    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440320    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440322    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440324    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440327    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440330    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440333    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.442555 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440335    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440338    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440340    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440343    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440346    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440348    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440351    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440353    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440356    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440358    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440361    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440363    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440366    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440368    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440371    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440373    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440376    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440379    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440382    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440384    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.443082 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440387    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440390    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440393    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440396    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440398    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440400    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440403    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440405    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440408    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440411    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440413    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440417    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440419    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440422    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440425    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440427    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440430    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440432    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440435    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440437    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.443596 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440440    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440443    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440445    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440449    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440453    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440456    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440459    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440462    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440465    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440468    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440471    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440473    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440476    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440483    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440487    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440971    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440976    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440979    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440982    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.444097 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440985    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440988    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440991    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440994    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.440997    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441000    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441002    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441005    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441007    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441011    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441013    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441016    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441019    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441021    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441024    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441027    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441029    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441032    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441034    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.444573 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441037    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441039    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441042    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441044    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441047    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441049    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441052    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441057    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441061    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441064    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441067    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441070    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441072    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441075    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441077    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441080    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441084    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441087    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441089    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441092    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.445058 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441094    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441097    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441100    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441102    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441105    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441108    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441111    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441113    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441116    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441119    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441121    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441124    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441127    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441129    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441132    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441134    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441137    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441139    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441142    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441145    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.445755 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441148    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441150    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441153    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441155    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441158    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441160    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441163    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441166    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441170    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441172    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441175    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441177    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441180    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441182    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441185    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441187    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441190    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441192    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441194    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441197    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.446592 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441200    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441202    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441204    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441207    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441209    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441228    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441232    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441235    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441239    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441241    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441244    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441247    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441250    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441252    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441256    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441260    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441264    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441266    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441269    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.447092 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441272    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441275    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441278    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.441282    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441372    4377 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441381    4377 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441389    4377 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441400    4377 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441407    4377 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441411    4377 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441415    4377 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441419    4377 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441423    4377 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441431    4377 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441435    4377 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441438    4377 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441441    4377 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441445    4377 flags.go:64] FLAG: --cgroup-root=""
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441447    4377 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441451    4377 flags.go:64] FLAG: --client-ca-file=""
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441454    4377 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441457    4377 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441462    4377 flags.go:64] FLAG: --cluster-domain=""
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441465    4377 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 02:55:00.447581 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441468    4377 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441471    4377 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441475    4377 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441479    4377 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441482    4377 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441486    4377 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441489    4377 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441492    4377 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441495    4377 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441499    4377 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441502    4377 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441505    4377 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441509    4377 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441512    4377 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441515    4377 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441540    4377 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441543    4377 flags.go:64] FLAG: --enable-server="true"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441547    4377 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441551    4377 flags.go:64] FLAG: --event-burst="100"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441554    4377 flags.go:64] FLAG: --event-qps="50"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441557    4377 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441560    4377 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441564    4377 flags.go:64] FLAG: --eviction-hard=""
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441568    4377 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 02:55:00.448166 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441571    4377 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441575    4377 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441578    4377 flags.go:64] FLAG: --eviction-soft=""
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441581    4377 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441584    4377 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441588    4377 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441591    4377 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441594    4377 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441597    4377 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441600    4377 flags.go:64] FLAG: --feature-gates=""
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441604    4377 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441607    4377 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441611    4377 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441614    4377 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441617    4377 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441620    4377 flags.go:64] FLAG: --help="false"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441622    4377 flags.go:64] FLAG: --hostname-override="ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441626    4377 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441628    4377 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441631    4377 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441635    4377 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441638    4377 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441641    4377 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 02:55:00.448804 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441644    4377 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441646    4377 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441649    4377 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441652    4377 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441655    4377 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441658    4377 flags.go:64] FLAG: --kube-reserved=""
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441661    4377 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441664    4377 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441667    4377 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441670    4377 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441675    4377 flags.go:64] FLAG: --lock-file=""
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441679    4377 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441682    4377 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441685    4377 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441690    4377 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441693    4377 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441696    4377 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441699    4377 flags.go:64] FLAG: --logging-format="text"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441702    4377 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441706    4377 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441709    4377 flags.go:64] FLAG: --manifest-url=""
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441712    4377 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441717    4377 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441720    4377 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441723    4377 flags.go:64] FLAG: --max-pods="110"
Oct 30 02:55:00.449374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441727    4377 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441730    4377 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441733    4377 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441736    4377 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441739    4377 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441742    4377 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441745    4377 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441752    4377 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441755    4377 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441758    4377 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441761    4377 flags.go:64] FLAG: --pod-cidr=""
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441764    4377 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441767    4377 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441770    4377 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441773    4377 flags.go:64] FLAG: --port="10250"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441776    4377 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441779    4377 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0006fa8e21ebc9997"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441784    4377 flags.go:64] FLAG: --qos-reserved=""
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441788    4377 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441792    4377 flags.go:64] FLAG: --register-node="true"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441794    4377 flags.go:64] FLAG: --register-with-taints=""
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441798    4377 flags.go:64] FLAG: --registry-burst="10"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441801    4377 flags.go:64] FLAG: --registry-qps="5"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441804    4377 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441807    4377 flags.go:64] FLAG: --reserved-memory=""
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441811    4377 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 02:55:00.449999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441814    4377 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441817    4377 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441820    4377 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441824    4377 flags.go:64] FLAG: --runonce="false"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441827    4377 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441831    4377 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441834    4377 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441837    4377 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441840    4377 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441843    4377 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441846    4377 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441849    4377 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441852    4377 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441855    4377 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441858    4377 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441861    4377 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441864    4377 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441867    4377 flags.go:64] FLAG: --system-cgroups=""
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441870    4377 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441876    4377 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441879    4377 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441882    4377 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441886    4377 flags.go:64] FLAG: --tls-min-version=""
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441894    4377 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441899    4377 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 02:55:00.450641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441909    4377 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441913    4377 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441916    4377 flags.go:64] FLAG: --v="2"
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441920    4377 flags.go:64] FLAG: --version="false"
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441925    4377 flags.go:64] FLAG: --vmodule=""
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441928    4377 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.441932    4377 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442067    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442071    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442074    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442077    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442081    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442099    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442103    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442106    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442110    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442113    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442116    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442119    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442122    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442124    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.451245 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442127    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442130    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442133    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442135    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442138    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442141    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442143    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442146    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442150    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442155    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442159    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442162    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442166    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442169    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442171    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442174    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442176    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442179    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442181    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.451847 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442184    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442186    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442189    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442192    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442195    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442197    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442200    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442202    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442205    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442208    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442210    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442212    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442215    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442217    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442220    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442222    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442225    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442227    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442230    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442232    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.452425 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442236    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442238    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442242    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442246    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442250    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442254    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442256    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442259    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442262    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442264    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442267    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442269    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442272    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442275    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442277    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442287    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442290    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442293    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442295    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.453073 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442298    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442301    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442303    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442306    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442308    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442311    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442313    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442316    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442318    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442321    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442324    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442326    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442329    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442331    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442334    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442338    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442342    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442345    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442348    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442350    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.453591 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442353    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442355    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442358    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442360    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442363    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442365    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442368    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442371    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442373    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442376    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442379    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442381    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442384    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442387    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.442414    4377 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442449    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.454121 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442454    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442457    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442459    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442462    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442464    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442467    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442470    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442472    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442475    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442478    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442481    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442485    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442487    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442492    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442495    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442497    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442500    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442503    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442505    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442508    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.454535 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442510    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442513    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442516    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442530    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442534    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442538    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442542    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442545    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442549    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442551    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442554    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442557    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442560    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442562    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442565    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442568    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442571    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442573    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442576    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.455162 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442579    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442581    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442584    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442586    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442590    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442593    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442597    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442601    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442603    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442606    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442609    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442611    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442614    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442616    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442619    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442622    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442625    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442627    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442630    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.455756 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442632    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442636    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442640    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442644    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442647    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442650    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442653    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442655    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442658    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442661    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442663    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442666    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442668    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442671    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442674    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442676    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442679    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442683    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442685    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.456282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442689    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442692    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442695    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442697    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442700    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442702    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442705    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442714    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442717    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442720    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442722    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442725    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442728    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442730    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442733    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442736    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442738    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442744    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442747    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442750    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442752    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.456772 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442755    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442757    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442760    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442763    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442765    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442768    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.442771    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.447730    4377 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.447749    4377 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.447769    4377 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.447774    4377 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447834    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447839    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447842    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447846    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447849    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447852    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447855    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447858    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447861    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.457294 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447864    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447866    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447869    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447872    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447875    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447878    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447881    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447884    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447887    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447889    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447892    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447894    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447897    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447900    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447903    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447905    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447908    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447911    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447914    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447917    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.457831 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447920    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447923    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447925    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447928    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447931    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447933    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447936    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447938    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447941    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447943    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447946    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447948    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447951    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447953    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447956    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447958    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447961    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447965    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447968    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.458327 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447971    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447974    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447976    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447979    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447981    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447985    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447987    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447990    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447992    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447995    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.447998    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448000    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448003    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448006    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448008    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448011    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448013    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448016    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448018    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.458785 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448021    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448023    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448026    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448030    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448034    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448037    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448040    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448043    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448045    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448048    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448051    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448054    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448057    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448060    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448062    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448066    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448070    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448072    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448075    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.459240 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448078    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448080    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448083    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448085    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448088    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448091    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448093    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448096    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448098    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448101    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448104    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448106    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448109    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448112    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448114    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448117    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448120    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448122    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448124    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.459721 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448127    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.448154    4377 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448291    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448295    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448298    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448301    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448304    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448307    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448310    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448312    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448315    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448318    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448321    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448324    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448326    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448328    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.460181 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448331    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448334    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448336    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448338    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448341    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448344    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448347    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448349    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448352    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448354    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448357    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448359    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448362    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448364    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448367    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448371    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448374    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448377    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448379    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.460852 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448382    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448385    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448387    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448389    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448392    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448395    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448398    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448401    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448403    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448405    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448408    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448411    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448413    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448416    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448418    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448421    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448423    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448426    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448429    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448431    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.461669 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448433    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448436    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448439    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448441    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448444    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448446    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448449    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448451    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448454    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448456    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448459    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448461    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448464    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448466    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448469    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448471    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448474    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448477    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448482    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448485    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.462408 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448487    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448490    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448492    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448495    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448498    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448500    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448502    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448505    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448508    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448510    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448513    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448515    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448533    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448538    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448542    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448545    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448548    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448551    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448553    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.463157 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448556    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448559    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448561    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448564    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448567    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448569    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448572    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448575    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448578    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448581    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448584    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448590    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448594    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448596    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.448621    4377 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 02:55:00.463652 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448653    4377 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448657    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448661    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448664    4377 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448666    4377 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448669    4377 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448672    4377 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448674    4377 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448677    4377 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448679    4377 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448682    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448685    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448688    4377 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448690    4377 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448693    4377 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448695    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448698    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448701    4377 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448703    4377 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 02:55:00.464025 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448706    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448708    4377 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448711    4377 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448715    4377 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448718    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448721    4377 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448723    4377 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448726    4377 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448728    4377 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448735    4377 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448737    4377 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448741    4377 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448743    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448746    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448748    4377 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448751    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448754    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448756    4377 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448759    4377 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 02:55:00.464495 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448762    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448764    4377 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448767    4377 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448769    4377 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448772    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448774    4377 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448777    4377 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448780    4377 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448782    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448785    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448788    4377 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448790    4377 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448793    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448796    4377 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448798    4377 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448801    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448803    4377 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448806    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448808    4377 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448813    4377 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 02:55:00.464965 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448817    4377 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448820    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448825    4377 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448828    4377 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448831    4377 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448839    4377 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448843    4377 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448845    4377 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448848    4377 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448851    4377 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448853    4377 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448856    4377 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448859    4377 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448861    4377 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448864    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448867    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448870    4377 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448872    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448875    4377 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448877    4377 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 02:55:00.465437 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448880    4377 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448883    4377 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448885    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448888    4377 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448890    4377 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448893    4377 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448896    4377 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448898    4377 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448901    4377 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448903    4377 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448906    4377 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448908    4377 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448911    4377 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448913    4377 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448918    4377 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448921    4377 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448923    4377 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448926    4377 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448932    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448938    4377 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 02:55:00.465939 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448943    4377 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448947    4377 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448952    4377 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448957    4377 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448960    4377 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448963    4377 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448965    4377 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:00.448968    4377 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.449026    4377 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.450000    4377 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.450059    4377 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.450237    4377 server.go:1014] "Starting client certificate rotation"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.450378    4377 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.450434    4377 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:49:21 +0000 UTC" deadline="2028-04-06 11:02:40.047969323 +0000 UTC"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.450464    4377 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12584h7m39.597507694s"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.450708    4377 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:55:00.466429 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.451880    4377 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 02:55:00.466976 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.453283    4377 log.go:25] "Validated CRI v1 runtime API"
Oct 30 02:55:00.466976 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.454740    4377 log.go:25] "Validated CRI v1 image API"
Oct 30 02:55:00.466976 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.455473    4377 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 02:55:00.466976 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.456965    4377 fs.go:135] Filesystem UUIDs: map[3142cb85-9a18-4850-b4c5-9f789ec9e743:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ae4c1181-9583-4106-bead-23f259c85786:/dev/nvme0n1p4]
Oct 30 02:55:00.467096 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.456987    4377 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c23ce0aafb5479699870ce8ce861e9ba09579e16ff6a1908a7de95481e70b33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c23ce0aafb5479699870ce8ce861e9ba09579e16ff6a1908a7de95481e70b33/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3df1ed6f31b6cdd5eac119537ee44ac3408de59e35818b3dcaef1217bfe85b0d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3df1ed6f31b6cdd5eac119537ee44ac3408de59e35818b3dcaef1217bfe85b0d/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/67573de29f7ffc278c446da4ea50d42488ff42b722e64eddcfff3ab4fbad3b72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/67573de29f7ffc278c446da4ea50d42488ff42b722e64eddcfff3ab4fbad3b72/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69d6b5f73f3955b2a76bfbdec7a1c110de46f4f30adf930a3a517b88e3aea820/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69d6b5f73f3955b2a76bfbdec7a1c110de46f4f30adf930a3a517b88e3aea820/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/736df77af67d2c9d88e61af84f6c786f8fc4da888f60ff02ba560cb4f0dfc72a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/736df77af67d2c9d88e61af84f6c786f8fc4da888f60ff02ba560cb4f0dfc72a/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8826e3848a5bc8c898216cb12d22dac13cb13c3ac1f1c1ae82f748420927f065/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8826e3848a5bc8c898216cb12d22dac13cb13c3ac1f1c1ae82f748420927f065/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a496efaf565d8e0d4f42c5c07863f46efa3165398d5336eb6a6ecaba850f3a9a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a496efaf565d8e0d4f42c5c07863f46efa3165398d5336eb6a6ecaba850f3a9a/userdata/shm major:0 minor:95 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/badf764c0fead9bee1750f6db27a4bac70c86f9d70cab2fa9e9eeb953752f6a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/badf764c0fead9bee1750f6db27a4bac70c86f9d70cab2fa9e9eeb953752f6a2/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd71a155762ae372acae0b06290c7e49a6b42beb7f6676287bb017fe649adbc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd71a155762ae372acae0b06290c7e49a6b42beb7f6676287bb017fe649adbc5/userdata/shm major:0 minor:99 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c4c951371de2e965f6faf8c75159dd5935825f634d130162fa7852b952797157/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c4c951371de2e965f6faf8c75159dd5935825f634d130162fa7852b952797157/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6bc118809ce20fcb0cf69f5b11c4acae308418b94bf8b8ab7fe72e86a0d6298/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6bc118809ce20fcb0cf69f5b11c4acae308418b94bf8b8ab7fe72e86a0d6298/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/027c260d-eae4-4dda-834e-be9433d32534/volumes/kubernetes.io~projected/kube-api-access-hxwkm:{mountpoint:/var/lib/kubelet/pods/027c260d-eae4-4dda-834e-be9433d32534/volumes/kubernetes.io~projected/kube-api-access-hxwkm major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09ab82b8-78fe-4884-9776-be21a025ec67/volumes/kubernetes.io~projected/kube-api-access-k7bgb:{mountpoint:/var/lib/kubelet/pods/09ab82b8-78fe-4884-9776-be21a025ec67/volumes/kubernetes.io~projected/kube-api-access-k7bgb major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f/volumes/kubernetes.io~projected/kube-api-access-r6f47:{mountpoint:/var/lib/kubelet/pods/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f/volumes/kubernetes.io~projected/kube-api-access-r6f47 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volumes/kubernetes.io~projected/kube-api-access-bppxt:{mountpoint:/var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volumes/kubernetes.io~projected/kube-api-access-bppxt major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~projected/kube-api-access-sqk7b:{mountpoint:/var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~projected/kube-api-access-sqk7b major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ef68d50-d2d2-4244-848f-208cf21b039a/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/3ef68d50-d2d2-4244-848f-208cf21b039a/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c/volumes/kubernetes.io~projected/kube-api-access-4jh6m:{mountpoint:/var/lib/kubelet/pods/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c/volumes/kubernetes.io~projected/kube-api-access-4jh6m major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/453941ab-5ec2-4b68-9991-492690f822be/volumes/kubernetes.io~projected/kube-api-access-rrqvb:{mountpoint:/var/lib/kubelet/pods/453941ab-5ec2-4b68-9991-492690f822be/volumes/kubernetes.io~projected/kube-api-access-rrqvb major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c4663c2-fffd-489d-8f4b-aa26181bff45/volumes/kubernetes.io~projected/kube-api-access-7s9cq:{mountpoint:/var/lib/kubelet/pods/6c4663c2-fffd-489d-8f4b-aa26181bff45/volumes/kubernetes.io~projected/kube-api-access-7s9cq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c4663c2-fffd-489d-8f4b-aa26181bff45/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/6c4663c2-fffd-489d-8f4b-aa26181bff45/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9fbf954b-1744-434a-8f0f-7ac3d546e081/volumes/kubernetes.io~projected/kube-api-access-98nz7:{mountpoint:/var/lib/kubelet/pods/9fbf954b-1744-434a-8f0f-7ac3d546e081/volumes/kubernetes.io~projected/kube-api-access-98nz7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccbc2f47-f780-45bf-8d91-e2d081bab080/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ccbc2f47-f780-45bf-8d91-e2d081bab080/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2252025-152e-4341-9a75-285b38cc2436/volumes/kubernetes.io~projected/kube-api-access-t27bz:{mountpoint:/var/lib/kubelet/pods/e2252025-152e-4341-9a75-285b38cc2436/volumes/kubernetes.io~projected/kube-api-access-t27bz major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f106e28a-9a25-4d6b-a16f-add5d5591b60/volumes/kubernetes.io~projected/kube-api-access-hcng4:{mountpoint:/var/lib/kubelet/pods/f106e28a-9a25-4d6b-a16f-add5d5591b60/volumes/kubernetes.io~projected/kube-api-access-hcng4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa50191b-3520-4499-a385-a0e6a40c3da5/volumes/kubernetes.io~projected/kube-api-access-6zdcj:{mountpoint:/var/lib/kubelet/pods/fa50191b-3520-4499-a385-a0e6a40c3da5/volumes/kubernetes.io~projected/kube-api-access-6zdcj major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe685890-13f3-4526-921b-56a13e3dcea6/volumes/kubernetes.io~projected/kube-api-access-q8bjg:{mountpoint:/var/lib/kubelet/pods/fe685890-13f3-4526-921b-56a13e3dcea6/volumes/kubernetes.io~projected/kube-api-access-q8bjg major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/abdcda084a7562bbf942b70b1217c8b4e843970c0f7e50aa277271a3714babe5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/f9ba739cf52ab5d66e5704898af44545bd6f96e35a3a421b24f93d34fc4c3656/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/166224eb6435fc77eabc8b2c8d4f2a9d3f232f93117b6f6e44d866819a3c38c2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/701e78cf177e47ba7ca9434308722b1df5e383d03501e15bbcf6a7c4c48e8fdf/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/77f9b00800898a77a590727c83c523aea5ae5bb9f4cb4aaba033f3237d0cc0f5/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/78e330d3fba7d74bca095dce4a2be3f3b8b8a1f63a6bece2b4cdc94d2fba9414/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/92cd57aefeaaefa7f1e80faea9a069d81e4808235a40f6d3b81edd1dca9a322a/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d996470e6a5357e345eab5328874c9272eed05391eacb75579d755b06fc09fac/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/db565a30de4ea39b6e64bb846a0dbab3a60cecb184c3c2eabcc3e18801f93db2/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/9da7e65e1f11ee735fe3fd2deed70035cd6bfc655dd204e73fc2037a3c89007e/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/e621f32bebe2f4c28bde275780eb814c0fe2cbc224a1791e40ee4f8747aafbf8/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/163607ac3c10454ca0ac563de2faa327ae32c72d5fdd3c640a082fac0a2b68eb/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/0c8cc1d3c394080c7402bfb77653e017d648fbc949fef4df8f5c7ef52a6ab3e7/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/b720e0dffdd6455d3a2c59886b3fa1f3075da97a043d387fe14fbd06e21d6aa8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/509befe187c1e608679c01fc1e75d939e8a4c6464f82f9dcb331143df8373a13/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9a6174de81b15285dff6507ad2b0edf878f111806cd958fbfce72a8dde427d24/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/0d3a3479fbe96d207ec740b0c9fab88690a09e02673f562706693f968520fc44/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3d1dfc45e3a1634764257cf8817afe096dfaa2817222be6b8f9b9141fbda62ad/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e3a3aecf1c22f974a30206f5e31f78c82952423c6f39dbceb4f1149707f089f1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5cbdeba7dc63cb593d997b56b58f592949bdba83d4c8e21fcbae526bab554755/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/95daf0a996a3cf5f5e4755b88edfb4e90b146e7c3450c386c81e4068b2759335/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/b563324c831357f8d8279ac84b05e3d5184b24a44ca2a92f0e6e677430c4f5c4/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/f73c855c892d176deeb1dacfffc8c1455732a7a2224e674b41ac97687307d06b/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-221:{mountpoint:/var/lib/containers/storage/overlay/1c1a8f617136ff81ac74b533513aed7aeec0e1b2c28ab0823194eed4afeac6a1/merged major:0 minor:221 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/16a289929f60f5716cf645459e3fcc67d7a32260c383ace16dc7fa587a06d7ed/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/70dcaf7c08c2db3de9e550a33d271a9a0883e544d562a9d8531ea5183449160b/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/b483928c2d6c20ca516ad67a8b2d09a40401f6103433730ca85cc164349b62ec/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bf73aa7291e0e46f2f86cd346d546de799ea6e3a02dde4504360a11e453d06db/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/91fc0857004c9be68eeecd2b863fb6b20742abb57ede9dc19481ff12dc0e7516/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/e675b8f558ef9dfaf3635b3bcb1812f67a956f3377d77c6b9f2512afffe6d675/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/bd10446fe2faa47ae2dd26a56d282dfbf791bb28f08e8b3a447976fdc26a42f1/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/66474a0e5ccbb7f63fa36c7001910a97386430651c9337add98a4fd97ae08510/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/3dc7bb4f4cd0a329fcd6503e6013b4ce3ca326b38930e3fa9f444bd29e0f386e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/60cb86e8b3fbb1bb74e536ea1ca8141262e8c83c946d36ad85a5dccf9c36e75e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f0b3d403620ecfaf9fa68548b749dc4e24777d9a627129f92be64edca6feb6a0/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/a5c3dfdb14fd77f39a9e1eed0d4038e1f7497abf33851abad9f0f5d41a596472/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/60dd01519db40b9ef65002dd02dc19a566c5d8d02633a8f11d0b6f962d45d56a/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465195    4377 manager.go:217] Machine: {Timestamp:2026-10-30 02:55:00.464715271 +0000 UTC m=+0.057044069 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3085771 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a3e4fb1f3aeee7ce3c1742ee15fdf SystemUUID:ec2a3e4f-b1f3-aeee-7ce3-c1742ee15fdf BootID:687a6ca5-73a4-4842-acd3-4ae67797a993 Filesystems:[{Device:/var/lib/kubelet/pods/3ef68d50-d2d2-4244-848f-208cf21b039a/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/67573de29f7ffc278c446da4ea50d42488ff42b722e64eddcfff3ab4fbad3b72/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/453941ab-5ec2-4b68-9991-492690f822be/volumes/kubernetes.io~projected/kube-api-access-rrqvb DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8826e3848a5bc8c898216cb12d22dac13cb13c3ac1f1c1ae82f748420927f065/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd71a155762ae372acae0b06290c7e49a6b42beb7f6676287bb017fe649adbc5/userdata/shm DeviceMajor:0 DeviceMinor:99 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ccbc2f47-f780-45bf-8d91-e2d081bab080/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:209 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~projected/kube-api-access-sqk7b DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c23ce0aafb5479699870ce8ce861e9ba09579e16ff6a1908a7de95481e70b33/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c/volumes/kubernetes.io~projected/kube-api-access-4jh6m DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe685890-13f3-4526-921b-56a13e3dcea6/volumes/kubernetes.io~projected/kube-api-access-q8bjg DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f/volumes/kubernetes.io~projected/kube-api-access-r6f47 DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c4c951371de2e965f6faf8c75159dd5935825f634d130162fa7852b952797157/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c4663c2-fffd-489d-8f4b-aa26181bff45/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3df1ed6f31b6cdd5eac119537ee44ac3408de59e35818b3dcaef1217bfe85b0d/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-221 DeviceMajor:0 DeviceMinor:221 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/736df77af67d2c9d88e61af84f6c786f8fc4da888f60ff02ba560cb4f0dfc72a/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/badf764c0fead9bee1750f6db27a4bac70c86f9d70cab2fa9e9eeb953752f6a2/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f106e28a-9a25-4d6b-a16f-add5d5591b60/volumes/kubernetes.io~projected/kube-api-access-hcng4 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa50191b-3520-4499-a385-a0e6a40c3da5/volumes/kubernetes.io~projected/kube-api-access-6zdcj DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/09ab82b8-78fe-4884-9776-be21a025ec67/volumes/kubernetes.io~projected/kube-api-access-k7bgb DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/27151663-70b9-4779-9730-fd65ee35bfcb/volumes/kubernetes.io~projected/kube-api-access-bppxt DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69d6b5f73f3955b2a76bfbdec7a1c110de46f4f30adf930a3a517b88e3aea820/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6bc118809ce20fcb0cf69f5b11c4acae308418b94bf8b8ab7fe72e86a0d6298/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2252025-152e-4341-9a75-285b38cc2436/volumes/kubernetes.io~projected/kube-api-access-t27bz DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6c4663c2-fffd-489d-8f4b-aa26181bff45/volumes/kubernetes.io~projected/kube-api-access-7s9cq DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a496efaf565d8e0d4f42c5c07863f46efa3165398d5336eb6a6ecaba850f3a9a/userdata/shm DeviceMajor:0 DeviceMinor:95 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9fbf954b-1744-434a-8f0f-7ac3d546e081/volumes/kubernetes.io~projected/kube-api-access-98nz7 DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/027c260d-eae4-4dda-834e-be9433d32534/volumes/kubernetes.io~projected/kube-api-access-hxwkm DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:3df1ed6f31b6cdd MacAddress:4e:db:49:de:5a:e3 Speed:10000 Mtu:8901} {Name:67573de29f7ffc2 MacAddress:62:3b:ec:34:04:90 Speed:10000 Mtu:8901} {Name:69d6b5f73f3955b MacAddress:6a:6a:ad:aa:01:34 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f7:51:c5:07 Speed:0 Mtu:9001} {Name:br-int MacAddress:92:69:26:91:fb:94 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:f7:51:c5:07 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ca:51:d7:ee:22:7d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:1e:ed:cd:a0:42:48 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465447    4377 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465495    4377 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465789    4377 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465812    4377 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-200.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465941    4377 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465949    4377 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465960    4377 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.465974    4377 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466077    4377 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466237    4377 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466351    4377 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466392    4377 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466402    4377 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466412    4377 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466421    4377 kubelet.go:405] "Adding apiserver pod source"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.466429    4377 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.467087    4377 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:55:00.467232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.467099    4377 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:55:00.467890 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.467633    4377 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 02:55:00.469430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469417    4377 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 02:55:00.469430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469429    4377 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 02:55:00.469540 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469530    4377 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 02:55:00.469729 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469718    4377 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 02:55:00.469762 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469747    4377 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 02:55:00.469847 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469839    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 02:55:00.469877 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469853    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 02:55:00.469877 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469859    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 02:55:00.469877 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469865    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 02:55:00.469877 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469871    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469878    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469885    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469891    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469898    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469903    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469917    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469927    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469951    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 02:55:00.469975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.469957    4377 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 02:55:00.473186 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.473174    4377 server.go:1263] "Started kubelet"
Oct 30 02:55:00.473596 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.473423    4377 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 02:55:00.476788 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.473477    4377 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 02:55:00.476788 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.473692    4377 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 02:55:00.476788 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.474391    4377 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 02:55:00.476788 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.476305    4377 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 02:55:00.473778 ip-10-0-138-200 systemd[1]: Started Kubernetes Kubelet.
Oct 30 02:55:00.477439 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.477422    4377 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.478067    4377 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.478248    4377 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:49:23 +0000 UTC" deadline="2028-04-02 15:09:11.08368122 +0000 UTC"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.478293    4377 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12492h14m10.605392332s"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.478600    4377 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.478612    4377 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.478770    4377 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.481359    4377 factory.go:55] Registering systemd factory
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.481380    4377 factory.go:223] Registration of the systemd container factory successfully
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.481895    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2252025-152e-4341-9a75-285b38cc2436" volumeName="kubernetes.io/projected/e2252025-152e-4341-9a75-285b38cc2436-kube-api-access-t27bz" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.481939    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f106e28a-9a25-4d6b-a16f-add5d5591b60" volumeName="kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:00.481940    4377 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-138-200.ec2.internal\" not found"
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.481959    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f106e28a-9a25-4d6b-a16f-add5d5591b60" volumeName="kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482018    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f106e28a-9a25-4d6b-a16f-add5d5591b60" volumeName="kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482098    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe685890-13f3-4526-921b-56a13e3dcea6" volumeName="kubernetes.io/projected/fe685890-13f3-4526-921b-56a13e3dcea6-kube-api-access-q8bjg" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482135    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="027c260d-eae4-4dda-834e-be9433d32534" volumeName="kubernetes.io/empty-dir/027c260d-eae4-4dda-834e-be9433d32534-data-volume" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482164    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09ab82b8-78fe-4884-9776-be21a025ec67" volumeName="kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482177    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ef68d50-d2d2-4244-848f-208cf21b039a" volumeName="kubernetes.io/secret/3ef68d50-d2d2-4244-848f-208cf21b039a-agent-certs" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482217    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccbc2f47-f780-45bf-8d91-e2d081bab080" volumeName="kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482238    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f106e28a-9a25-4d6b-a16f-add5d5591b60" volumeName="kubernetes.io/projected/f106e28a-9a25-4d6b-a16f-add5d5591b60-kube-api-access-hcng4" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482250    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e0bfb91-a5ed-41c0-a26e-fe66447eda3b" volumeName="kubernetes.io/projected/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-kube-api-access-sqk7b" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482346    4377 factory.go:153] Registering CRI-O factory
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482374    4377 factory.go:223] Registration of the crio container factory successfully
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482454    4377 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482476    4377 factory.go:103] Registering Raw factory
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482494    4377 manager.go:1199] Started watching for new ooms in manager
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482267    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9fbf954b-1744-434a-8f0f-7ac3d546e081" volumeName="kubernetes.io/projected/9fbf954b-1744-434a-8f0f-7ac3d546e081-kube-api-access-98nz7" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482608    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2252025-152e-4341-9a75-285b38cc2436" volumeName="kubernetes.io/configmap/e2252025-152e-4341-9a75-285b38cc2436-config-volume" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482640    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27151663-70b9-4779-9730-fd65ee35bfcb" volumeName="kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-env-overrides" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482667    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b96a13-ca71-48c9-9fa1-6d785e9e3b3c" volumeName="kubernetes.io/configmap/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-serviceca" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482692    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa50191b-3520-4499-a385-a0e6a40c3da5" volumeName="kubernetes.io/projected/fa50191b-3520-4499-a385-a0e6a40c3da5-kube-api-access-6zdcj" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482734    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44b96a13-ca71-48c9-9fa1-6d785e9e3b3c" volumeName="kubernetes.io/projected/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-kube-api-access-4jh6m" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482750    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453941ab-5ec2-4b68-9991-492690f822be" volumeName="kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-multus-daemon-config" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482776    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c4663c2-fffd-489d-8f4b-aa26181bff45" volumeName="kubernetes.io/secret/6c4663c2-fffd-489d-8f4b-aa26181bff45-cert" seLinuxMountContext=""
Oct 30 02:55:00.482760 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482788    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27151663-70b9-4779-9730-fd65ee35bfcb" volumeName="kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482805    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27151663-70b9-4779-9730-fd65ee35bfcb" volumeName="kubernetes.io/secret/27151663-70b9-4779-9730-fd65ee35bfcb-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482819    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e0bfb91-a5ed-41c0-a26e-fe66447eda3b" volumeName="kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-tmp" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482850    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa50191b-3520-4499-a385-a0e6a40c3da5" volumeName="kubernetes.io/configmap/fa50191b-3520-4499-a385-a0e6a40c3da5-iptables-alerter-script" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482884    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f" volumeName="kubernetes.io/empty-dir/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-tmp-dir" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482898    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e0bfb91-a5ed-41c0-a26e-fe66447eda3b" volumeName="kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-tuned" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482918    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="027c260d-eae4-4dda-834e-be9433d32534" volumeName="kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482931    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="027c260d-eae4-4dda-834e-be9433d32534" volumeName="kubernetes.io/projected/027c260d-eae4-4dda-834e-be9433d32534-kube-api-access-hxwkm" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.482992    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f" volumeName="kubernetes.io/projected/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-kube-api-access-r6f47" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483021    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27151663-70b9-4779-9730-fd65ee35bfcb" volumeName="kubernetes.io/projected/27151663-70b9-4779-9730-fd65ee35bfcb-kube-api-access-bppxt" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483045    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453941ab-5ec2-4b68-9991-492690f822be" volumeName="kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-cni-binary-copy" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483095    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="453941ab-5ec2-4b68-9991-492690f822be" volumeName="kubernetes.io/projected/453941ab-5ec2-4b68-9991-492690f822be-kube-api-access-rrqvb" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483643    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c4663c2-fffd-489d-8f4b-aa26181bff45" volumeName="kubernetes.io/projected/6c4663c2-fffd-489d-8f4b-aa26181bff45-kube-api-access-7s9cq" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483660    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2252025-152e-4341-9a75-285b38cc2436" volumeName="kubernetes.io/empty-dir/e2252025-152e-4341-9a75-285b38cc2436-tmp-dir" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483681    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27151663-70b9-4779-9730-fd65ee35bfcb" volumeName="kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-config" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483694    4377 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ef68d50-d2d2-4244-848f-208cf21b039a" volumeName="kubernetes.io/configmap/3ef68d50-d2d2-4244-848f-208cf21b039a-konnectivity-ca" seLinuxMountContext=""
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483705    4377 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 02:55:00.484232 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.483713    4377 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 02:55:00.484868 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.484803    4377 manager.go:319] Starting recovery of all containers
Oct 30 02:55:00.507120 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.507057    4377 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 02:55:00.508505 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.508481    4377 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 02:55:00.508505 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.508509    4377 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 02:55:00.508851 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.508571    4377 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 02:55:00.508851 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:00.508612    4377 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 02:55:00.521175 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.521134    4377 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="d75ca9a673234451598fe81cd1f7fa75f82d84703c4370f58a34589f12bf1e67" exitCode=0
Oct 30 02:55:00.521272 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.521174    4377 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="0bd4092514a6e49bde34138aa29f2ee7ff0f7fbb2bce8b896947b2e772a3da99" exitCode=0
Oct 30 02:55:00.521272 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.521190    4377 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="6e7143e9417f6ed5663b45d66fc23e0edb58849f919f10e1597563a610f11628" exitCode=0
Oct 30 02:55:00.521272 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.521201    4377 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="cdc3eed5bf0faae97dd0743d3968ea5ad85e37a520b72b40237226272f9fe338" exitCode=0
Oct 30 02:55:00.521272 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.521212    4377 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="545de7aae91fe292b18fed27e51929ec3c13e2f75d381a0ec4423acb24a8c8de" exitCode=0
Oct 30 02:55:00.521272 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.521224    4377 generic.go:356] "Generic (PLEG): container finished" podID="f106e28a-9a25-4d6b-a16f-add5d5591b60" containerID="1d212cbbafcbd087ec1f3e7fe77a431c6a30e5267456219ebcc14a68198690ef" exitCode=0
Oct 30 02:55:00.533764 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.533730    4377 generic.go:356] "Generic (PLEG): container finished" podID="77d685204da0cc5fa69ed201b4cfe0a4" containerID="71bb6fc09d82a11d5c1e06bccc66b6912240e63ac0a27ea0d4217bd56ecbe15d" exitCode=0
Oct 30 02:55:00.542312 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.542288    4377 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:00.546632 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.546615    4377 manager.go:324] Recovery completed
Oct 30 02:55:00.553399 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.553377    4377 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:00.553559 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.553541    4377 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:00.555077 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.555063    4377 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 02:55:00.555886 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.555872    4377 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 02:55:00.561767 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561750    4377 cpu_manager.go:226] "Starting" policy="none"
Oct 30 02:55:00.561767 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561765    4377 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 02:55:00.561874 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561784    4377 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 02:55:00.561948 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561930    4377 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 02:55:00.561978 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561950    4377 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 02:55:00.561978 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561974    4377 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 02:55:00.562029 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561980    4377 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 02:55:00.562029 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561986    4377 policy_none.go:50] "Start"
Oct 30 02:55:00.562029 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.561995    4377 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 02:55:00.562029 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.562005    4377 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 02:55:00.562149 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.562139    4377 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 02:55:00.562178 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.562150    4377 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 02:55:00.562178 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.562155    4377 policy_none.go:44] "Start"
Oct 30 02:55:00.565983 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.565959    4377 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 02:55:00.569058 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.569047    4377 manager.go:341] "Starting Device Plugin manager"
Oct 30 02:55:00.569709 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:00.569068    4377 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 02:55:00.569763 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.569714    4377 server.go:87] "Starting device plugin registration server"
Oct 30 02:55:00.570038 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.570027    4377 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 02:55:00.570068 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.570044    4377 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 02:55:00.570149 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.570128    4377 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 02:55:00.570288 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.570228    4377 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 02:55:00.570288 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.570266    4377 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 02:55:00.570288 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.570272    4377 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 02:55:00.570436 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.570320    4377 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:55:00.570256077Z","UUID":"9edb3b06-6c62-4ba1-8183-b4133dead206","Handler":null,"Name":"","Endpoint":""}
Oct 30 02:55:00.571478 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:00.571450    4377 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 02:55:00.571850 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.571612    4377 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 02:55:00.571850 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.571642    4377 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 02:55:00.609789 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.609706    4377 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"]
Oct 30 02:55:00.610193 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.610163    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"d1b766764a60d5387ea6838359f9932270b971a5fcf845f89d0a4e6b28b701de"}
Oct 30 02:55:00.610252 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.610202    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal" event={"ID":"fd03e27bf6704eb151f6eb79c368f447","Type":"ContainerStarted","Data":"0261a44b5a4c5cecb0e171f5883504470040a35b7955d8c2d3e077bf22eac7b3"}
Oct 30 02:55:00.610316 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.610255    4377 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.610316 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.610265    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"e36441d99636c9836e8e5e086d29aa0768b1c7d2ed873ee09b1d8780425c5c4f"}
Oct 30 02:55:00.610316 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.610289    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerDied","Data":"71bb6fc09d82a11d5c1e06bccc66b6912240e63ac0a27ea0d4217bd56ecbe15d"}
Oct 30 02:55:00.610316 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.610306    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal" event={"ID":"77d685204da0cc5fa69ed201b4cfe0a4","Type":"ContainerStarted","Data":"a713c9cdcc9182752e2c482f72db1850fb9ea4b0b26a7901e1eb14cac91a62f9"}
Oct 30 02:55:00.610507 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.610444    4377 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.635819 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.635799    4377 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:00.635923 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:00.635836    4377 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.639803 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.639784    4377 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:00.639911 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:00.639832    4377 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.671111 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.671084    4377 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 02:55:00.672569 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.672548    4377 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 02:55:00.672647 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.672584    4377 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 02:55:00.672647 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.672595    4377 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-138-200.ec2.internal" event="NodeHasSufficientPID"
Oct 30 02:55:00.672647 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.672627    4377 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.708560 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.708535    4377 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.708635 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.708602    4377 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.757278 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.757247    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.757278 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.757283    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.757461 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.757302    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.857497 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.857464    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.857497 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.857498    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.857715 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.857531    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.857715 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.857574    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.857715 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.857584    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/77d685204da0cc5fa69ed201b4cfe0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" (UID: \"77d685204da0cc5fa69ed201b4cfe0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:00.857715 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:00.857584    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fd03e27bf6704eb151f6eb79c368f447-config\") pod \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" (UID: \"fd03e27bf6704eb151f6eb79c368f447\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:01.467048 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.467012    4377 apiserver.go:52] "Watching apiserver"
Oct 30 02:55:01.536189 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.536160    4377 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 02:55:01.536576 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.536559    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-mwh8j","openshift-dns/dns-default-h9c28","kube-system/konnectivity-agent-j9s8d","kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal","openshift-insights/insights-runtime-extractor-jtjnj","openshift-network-diagnostics/network-check-target-mrnst","openshift-network-operator/iptables-alerter-fttld","kube-system/global-pull-secret-syncer-lhtq6","openshift-dns/node-resolver-x8h8g","openshift-ingress-canary/ingress-canary-rq99w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f","openshift-image-registry/node-ca-zq2m7","openshift-multus/multus-additional-cni-plugins-6x562","openshift-multus/multus-5bwp8","openshift-multus/network-metrics-daemon-s258j","openshift-ovn-kubernetes/ovnkube-node-fvzlt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"]
Oct 30 02:55:01.538350 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.538324    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:01.543349 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.543330    4377 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:01.543462 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.543337    4377 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:01.544065 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.544049    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.544385 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.544369    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.554721 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.554701    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d6p9p\""
Oct 30 02:55:01.555044 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.555025    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 02:55:01.555044 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.555033    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vxw8k\""
Oct 30 02:55:01.555225 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.555211    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 02:55:01.555284 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.555260    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-f9m7f\""
Oct 30 02:55:01.555363 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.555291    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-sdsc6\""
Oct 30 02:55:01.555988 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.555974    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 02:55:01.556045 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556015    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cr2zp\""
Oct 30 02:55:01.556694 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556673    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.556799 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556784    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 02:55:01.556865 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556838    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.556865 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556847    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.556977 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556891    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 02:55:01.556977 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556902    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-l729x\""
Oct 30 02:55:01.556977 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556902    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 02:55:01.556977 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.556969    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-h9jxb\""
Oct 30 02:55:01.557148 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.557059    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 02:55:01.557148 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.557092    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 02:55:01.557148 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.557097    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 02:55:01.557148 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.557097    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 02:55:01.557148 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.557120    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.557148 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.557142    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.557368 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.557205    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.567532 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567495    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 02:55:01.567718 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567705    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 02:55:01.567796 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567751    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.567796 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567789    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 02:55:01.567892 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567827    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.567892 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567848    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.567892 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567853    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.568035 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.567977    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.568297 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568279    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.568297 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568280    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5k8rj\""
Oct 30 02:55:01.568414 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568322    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 02:55:01.568414 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568333    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tm8xk\""
Oct 30 02:55:01.568414 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568340    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vnlvm\""
Oct 30 02:55:01.568414 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568334    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.568572 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568530    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-z4s4b\""
Oct 30 02:55:01.568811 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568799    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 02:55:01.568850 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568835    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.568902 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568892    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rg86w\""
Oct 30 02:55:01.568936 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568922    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.568936 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568923    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.569005 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.568922    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.570639 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.570624    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4tlgk\""
Oct 30 02:55:01.571869 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.571856    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 02:55:01.571919 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.571911    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 02:55:01.572230 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572215    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 02:55:01.572266 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572233    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.572410 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572380    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qkhcx\""
Oct 30 02:55:01.572410 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572391    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 02:55:01.572410 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572401    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 02:55:01.572563 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572430    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 02:55:01.572563 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572451    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 02:55:01.572651 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.572599    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvbgx\""
Oct 30 02:55:01.575178 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.575162    4377 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:01.575224 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.575202    4377 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-138-200.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:01.576712 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.576698    4377 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 02:55:01.576775 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.576736    4377 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-200.ec2.internal"
Oct 30 02:55:01.580682 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.580664    4377 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 02:55:01.662435 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662402    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.662435 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662438    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-run\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.662645 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662461    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-conf-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.662645 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662482    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.662645 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662562    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98nz7\" (UniqueName: \"kubernetes.io/projected/9fbf954b-1744-434a-8f0f-7ac3d546e081-kube-api-access-98nz7\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.662645 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662592    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jh6m\" (UniqueName: \"kubernetes.io/projected/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-kube-api-access-4jh6m\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:55:01.662836 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662724    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252025-152e-4341-9a75-285b38cc2436-tmp-dir\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.662836 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662765    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-tmp-dir\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:55:01.662836 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662792    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-etc-kubernetes\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.662968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662841    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2252025-152e-4341-9a75-285b38cc2436-tmp-dir\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.662968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662900    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-kubelet\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.662968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662899    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-tmp-dir\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:55:01.662968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662934    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-slash\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.662968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.662962    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-device-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.663171 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663000    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6f47\" (UniqueName: \"kubernetes.io/projected/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-kube-api-access-r6f47\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:55:01.663171 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663054    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysconfig\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.663171 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663135    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-k8s-cni-cncf-io\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.663320 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663189    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrqvb\" (UniqueName: \"kubernetes.io/projected/453941ab-5ec2-4b68-9991-492690f822be-kube-api-access-rrqvb\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.663320 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663238    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27151663-70b9-4779-9730-fd65ee35bfcb-ovn-node-metrics-cert\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663320 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663265    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/027c260d-eae4-4dda-834e-be9433d32534-data-volume\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.663454 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663357    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/027c260d-eae4-4dda-834e-be9433d32534-data-volume\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.663454 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663373    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-serviceca\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:55:01.663454 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663421    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ef68d50-d2d2-4244-848f-208cf21b039a-agent-certs\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:55:01.663454 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663438    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/27151663-70b9-4779-9730-fd65ee35bfcb-ovn-node-metrics-cert\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663632 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663456    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252025-152e-4341-9a75-285b38cc2436-config-volume\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.663632 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663474    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-netns\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663632 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663500    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-var-lib-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663632 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663541    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-etc-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663632 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663579    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663643    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-config\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663662    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663715    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-cnibin\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663775    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s9cq\" (UniqueName: \"kubernetes.io/projected/6c4663c2-fffd-489d-8f4b-aa26181bff45-kube-api-access-7s9cq\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663782    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2252025-152e-4341-9a75-285b38cc2436-config-volume\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663803    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3ef68d50-d2d2-4244-848f-208cf21b039a-agent-certs\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663846    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-kubelet-config\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663871    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-system-cni-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663891    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-bin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663910    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-config\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.663995 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.663935    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-multus\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664010    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-multus-daemon-config\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664099    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-ovn\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664141    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-script-lib\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664202    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa50191b-3520-4499-a385-a0e6a40c3da5-iptables-alerter-script\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664226    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-serviceca\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664291    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-multus-daemon-config\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664315    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664403    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-socket-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.664430 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664425    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-ovnkube-script-lib\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664436    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-registration-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664467    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-hosts-file\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664534    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-tmp\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664566    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fa50191b-3520-4499-a385-a0e6a40c3da5-iptables-alerter-script\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664571    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664605    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-sys\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664647    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-tmp\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664754    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqk7b\" (UniqueName: \"kubernetes.io/projected/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-kube-api-access-sqk7b\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664783    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-system-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664818    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664841    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664862    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ccbc2f47-f780-45bf-8d91-e2d081bab080-original-pull-secret\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.664885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664882    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-log-socket\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.665386 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664924    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/027c260d-eae4-4dda-834e-be9433d32534-crio-socket\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.665386 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664966    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-env-overrides\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.665386 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.664999    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwkm\" (UniqueName: \"kubernetes.io/projected/027c260d-eae4-4dda-834e-be9433d32534-kube-api-access-hxwkm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.665386 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665093    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-host\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:55:01.665386 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665259    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/27151663-70b9-4779-9730-fd65ee35bfcb-env-overrides\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.665612 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665411    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcng4\" (UniqueName: \"kubernetes.io/projected/f106e28a-9a25-4d6b-a16f-add5d5591b60-kube-api-access-hcng4\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.665679 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665659    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-kubelet\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.665792 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665770    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-bin\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.665863 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665847    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bjg\" (UniqueName: \"kubernetes.io/projected/fe685890-13f3-4526-921b-56a13e3dcea6-kube-api-access-q8bjg\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:01.665938 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665921    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t27bz\" (UniqueName: \"kubernetes.io/projected/e2252025-152e-4341-9a75-285b38cc2436-kube-api-access-t27bz\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.666017 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.665959    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.666176 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666090    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-kubernetes\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.666245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666216    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-conf\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.666660 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666639    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.666752 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666734    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-systemd\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.666802 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666776    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-cni-binary-copy\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.666858 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666810    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-netns\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.666904 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666885    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-node-log\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.666954 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666917    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-dbus\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.666954 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666945    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.667043 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.666977    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-var-lib-kubelet\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.667043 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667018    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-tuned\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.667177 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667069    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-cnibin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.667419 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667398    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-systemd-units\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.667491 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667444    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bppxt\" (UniqueName: \"kubernetes.io/projected/27151663-70b9-4779-9730-fd65ee35bfcb-kube-api-access-bppxt\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.667491 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667480    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.667619 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667541    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.667619 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667572    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-lib-modules\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.667619 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667604    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-host\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.667775 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667629    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-systemd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.667775 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667659    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-sys-fs\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.667775 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667706    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa50191b-3520-4499-a385-a0e6a40c3da5-host-slash\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:55:01.667775 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667760    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zdcj\" (UniqueName: \"kubernetes.io/projected/fa50191b-3520-4499-a385-a0e6a40c3da5-kube-api-access-6zdcj\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:55:01.667967 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667794    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:55:01.667967 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667826    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.667967 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.667854    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-hostroot\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.668132 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668013    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-netd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.668132 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668045    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:01.668132 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668076    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4663c2-fffd-489d-8f4b-aa26181bff45-cert\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:55:01.668132 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668104    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ef68d50-d2d2-4244-848f-208cf21b039a-konnectivity-ca\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:55:01.668351 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668136    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.668351 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668166    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.668351 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668199    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-modprobe-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.668351 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668231    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-socket-dir-parent\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.668351 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668257    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-multus-certs\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.668717 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668696    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.668784 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668738    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/453941ab-5ec2-4b68-9991-492690f822be-cni-binary-copy\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.668784 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.668768    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-tuned\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.669135 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.669117    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4663c2-fffd-489d-8f4b-aa26181bff45-cert\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:55:01.669201 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.669134    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3ef68d50-d2d2-4244-848f-208cf21b039a-konnectivity-ca\") pod \"konnectivity-agent-j9s8d\" (UID: \"3ef68d50-d2d2-4244-848f-208cf21b039a\") " pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:55:01.669316 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.669300    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f106e28a-9a25-4d6b-a16f-add5d5591b60-cni-binary-copy\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.669367 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.669344    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/027c260d-eae4-4dda-834e-be9433d32534-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.685687 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.685667    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jh6m\" (UniqueName: \"kubernetes.io/projected/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-kube-api-access-4jh6m\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:55:01.685896 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.685879    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98nz7\" (UniqueName: \"kubernetes.io/projected/9fbf954b-1744-434a-8f0f-7ac3d546e081-kube-api-access-98nz7\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.685939 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.685906    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6f47\" (UniqueName: \"kubernetes.io/projected/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-kube-api-access-r6f47\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:55:01.687264 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.687249    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrqvb\" (UniqueName: \"kubernetes.io/projected/453941ab-5ec2-4b68-9991-492690f822be-kube-api-access-rrqvb\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.687323 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.687302    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqk7b\" (UniqueName: \"kubernetes.io/projected/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-kube-api-access-sqk7b\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.687420 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.687404    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s9cq\" (UniqueName: \"kubernetes.io/projected/6c4663c2-fffd-489d-8f4b-aa26181bff45-kube-api-access-7s9cq\") pod \"ingress-canary-rq99w\" (UID: \"6c4663c2-fffd-489d-8f4b-aa26181bff45\") " pod="openshift-ingress-canary/ingress-canary-rq99w"
Oct 30 02:55:01.694136 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.694113    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zdcj\" (UniqueName: \"kubernetes.io/projected/fa50191b-3520-4499-a385-a0e6a40c3da5-kube-api-access-6zdcj\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:55:01.694257 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.694116    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bppxt\" (UniqueName: \"kubernetes.io/projected/27151663-70b9-4779-9730-fd65ee35bfcb-kube-api-access-bppxt\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.694257 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.694235    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcng4\" (UniqueName: \"kubernetes.io/projected/f106e28a-9a25-4d6b-a16f-add5d5591b60-kube-api-access-hcng4\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.694333 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.694309    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bjg\" (UniqueName: \"kubernetes.io/projected/fe685890-13f3-4526-921b-56a13e3dcea6-kube-api-access-q8bjg\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:01.695947 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.695931    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t27bz\" (UniqueName: \"kubernetes.io/projected/e2252025-152e-4341-9a75-285b38cc2436-kube-api-access-t27bz\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.696245 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.696230    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwkm\" (UniqueName: \"kubernetes.io/projected/027c260d-eae4-4dda-834e-be9433d32534-kube-api-access-hxwkm\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.696668 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.696653    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bgb\" (UniqueName: \"kubernetes.io/projected/09ab82b8-78fe-4884-9776-be21a025ec67-kube-api-access-k7bgb\") pod \"network-check-target-mrnst\" (UID: \"09ab82b8-78fe-4884-9776-be21a025ec67\") " pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:55:01.768970 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.768901    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-cnibin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.768970 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.768931    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-systemd-units\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.768970 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.768949    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.769141 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.768972    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-lib-modules\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769141 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.768993    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-host\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769141 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769034    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-cnibin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769141 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769037    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-systemd-units\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769141 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769047    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-host\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769141 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769119    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-lib-modules\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769117    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-systemd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769156    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-systemd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769178    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-sys-fs\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769199    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa50191b-3520-4499-a385-a0e6a40c3da5-host-slash\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769214    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769227    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-hostroot\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769246    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-netd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769267    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769275    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fa50191b-3520-4499-a385-a0e6a40c3da5-host-slash\") pod \"iptables-alerter-fttld\" (UID: \"fa50191b-3520-4499-a385-a0e6a40c3da5\") " pod="openshift-network-operator/iptables-alerter-fttld"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769282    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-hostroot\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769285    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.769324 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769320    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-modprobe-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769346    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-socket-dir-parent\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769353    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-sys-fs\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769359    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-netd\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769324    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769370    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-multus-certs\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769388    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769394    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-multus-certs\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769396    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769405    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-socket-dir-parent\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769420    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-run\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769443    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-conf-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769461    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769459    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-modprobe-d\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769505    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-etc-kubernetes\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.769502    4377 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769506    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-conf-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769549    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769559    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-kubelet\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.769583    4377 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.769592    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:02.269577207 +0000 UTC m=+1.861906016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769595    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-kubelet\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769605    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-slash\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769606    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-run\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769628    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-device-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.769642    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:02.269627053 +0000 UTC m=+1.861955840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : secret "metrics-daemon-secret" not found
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769641    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-etc-kubernetes\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769652    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-slash\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769671    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-device-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769681    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysconfig\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769718    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-k8s-cni-cncf-io\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769738    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysconfig\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769749    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-netns\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769772    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-k8s-cni-cncf-io\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769775    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-var-lib-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769784    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-netns\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769799    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-etc-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769825    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769839    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-var-lib-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769852    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769867    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.769853 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769870    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-etc-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769899    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-cnibin\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769927    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-kubelet-config\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.769938    4377 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769951    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-system-cni-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769957    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-cnibin\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:01.769971    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:02.269962082 +0000 UTC m=+1.862291101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769983    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-bin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.769994    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-kubelet-config\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770001    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-multus\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770005    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f106e28a-9a25-4d6b-a16f-add5d5591b60-system-cni-dir\") pod \"multus-additional-cni-plugins-6x562\" (UID: \"f106e28a-9a25-4d6b-a16f-add5d5591b60\") " pod="openshift-multus/multus-additional-cni-plugins-6x562"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770026    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-bin\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770033    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-cni-multus\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770035    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-ovn\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770058    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770066    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-ovn\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770074    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-socket-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770087    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-run-ovn-kubernetes\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770088    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-registration-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770116    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-hosts-file\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770120    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-registration-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770134    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-sys\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770149    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-system-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770155    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f-hosts-file\") pod \"node-resolver-x8h8g\" (UID: \"0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f\") " pod="openshift-dns/node-resolver-x8h8g"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770166    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770131    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-socket-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770201    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-sys\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770215    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-system-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770228    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-multus-cni-dir\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770244    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770264    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-log-socket\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770298    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/027c260d-eae4-4dda-834e-be9433d32534-crio-socket\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770308    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-run-openvswitch\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770322    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-host\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770334    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-log-socket\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770346    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-kubelet\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770360    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/027c260d-eae4-4dda-834e-be9433d32534-crio-socket\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770371    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-bin\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770396    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-kubernetes\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770398    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-var-lib-kubelet\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770401    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-host-cni-bin\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770417    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-conf\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770372    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/44b96a13-ca71-48c9-9fa1-6d785e9e3b3c-host\") pod \"node-ca-zq2m7\" (UID: \"44b96a13-ca71-48c9-9fa1-6d785e9e3b3c\") " pod="openshift-image-registry/node-ca-zq2m7"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770427    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-kubernetes\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770432    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-systemd\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770458    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-netns\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770479    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-node-log\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770489    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-systemd\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770501    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-dbus\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770529    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-var-lib-kubelet\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770540    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/453941ab-5ec2-4b68-9991-492690f822be-host-run-netns\") pod \"multus-5bwp8\" (UID: \"453941ab-5ec2-4b68-9991-492690f822be\") " pod="openshift-multus/multus-5bwp8"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770543    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/27151663-70b9-4779-9730-fd65ee35bfcb-node-log\") pod \"ovnkube-node-fvzlt\" (UID: \"27151663-70b9-4779-9730-fd65ee35bfcb\") " pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770559    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-etc-sysctl-conf\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770588    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ccbc2f47-f780-45bf-8d91-e2d081bab080-dbus\") pod \"global-pull-secret-syncer-lhtq6\" (UID: \"ccbc2f47-f780-45bf-8d91-e2d081bab080\") " pod="kube-system/global-pull-secret-syncer-lhtq6"
Oct 30 02:55:01.771209 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.770594    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0bfb91-a5ed-41c0-a26e-fe66447eda3b-var-lib-kubelet\") pod \"tuned-mwh8j\" (UID: \"2e0bfb91-a5ed-41c0-a26e-fe66447eda3b\") " pod="openshift-cluster-node-tuning-operator/tuned-mwh8j"
Oct 30 02:55:01.784763 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:01.784745    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9fbf954b-1744-434a-8f0f-7ac3d546e081-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vv8f\" (UID: \"9fbf954b-1744-434a-8f0f-7ac3d546e081\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vv8f"
Oct 30 02:55:02.274098 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:02.274060    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:02.274098 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:02.274103    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:02.274249 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:02.274130    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:02.274249 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:02.274217    4377 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:02.274249 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:02.274221    4377 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:02.274368 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:02.274269    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.274257194 +0000 UTC m=+2.866585978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:02.274368 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:02.274221    4377 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:02.274368 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:02.274330    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.274311083 +0000 UTC m=+2.866639877 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:02.274368 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:02.274357    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:03.274346013 +0000 UTC m=+2.866674801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : secret "metrics-daemon-secret" not found
Oct 30 02:55:02.914820 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:02.914792    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:03.281626 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:03.281593    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:03.281795 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:03.281634    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:03.281795 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:03.281660    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:03.281795 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:03.281757    4377 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:03.281905 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:03.281815    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls podName:027c260d-eae4-4dda-834e-be9433d32534 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:05.281802913 +0000 UTC m=+4.874131700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jtjnj" (UID: "027c260d-eae4-4dda-834e-be9433d32534") : secret "insights-runtime-extractor-tls" not found
Oct 30 02:55:03.281905 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:03.281754    4377 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 02:55:03.281905 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:03.281888    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls podName:e2252025-152e-4341-9a75-285b38cc2436 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:05.281874021 +0000 UTC m=+4.874202805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls") pod "dns-default-h9c28" (UID: "e2252025-152e-4341-9a75-285b38cc2436") : secret "dns-default-metrics-tls" not found
Oct 30 02:55:03.281905 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:03.281754    4377 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 02:55:03.282040 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:03.281922    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs podName:fe685890-13f3-4526-921b-56a13e3dcea6 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:05.281916422 +0000 UTC m=+4.874245206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs") pod "network-metrics-daemon-s258j" (UID: "fe685890-13f3-4526-921b-56a13e3dcea6") : secret "metrics-daemon-secret" not found
Oct 30 02:55:05.295295 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.295227    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:05.295821 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.295317    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:05.295821 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.295345    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:05.295821 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.295680    4377 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 02:55:05.297694 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.297671    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe685890-13f3-4526-921b-56a13e3dcea6-metrics-certs\") pod \"network-metrics-daemon-s258j\" (UID: \"fe685890-13f3-4526-921b-56a13e3dcea6\") " pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:05.297816 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.297793    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/027c260d-eae4-4dda-834e-be9433d32534-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jtjnj\" (UID: \"027c260d-eae4-4dda-834e-be9433d32534\") " pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:05.297859 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.297794    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2252025-152e-4341-9a75-285b38cc2436-metrics-tls\") pod \"dns-default-h9c28\" (UID: \"e2252025-152e-4341-9a75-285b38cc2436\") " pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:05.439216 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.439174    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s258j"
Oct 30 02:55:05.444323 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.444306    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:05.445406 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.445384    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jtjnj"
Oct 30 02:55:05.744618 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.744539    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s258j"]
Oct 30 02:55:05.747338 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:05.747303    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe685890_13f3_4526_921b_56a13e3dcea6.slice/crio-ddee0bbc680d7b2fd81d2ace2fe52e9aae4a3e8e51e565be5f45477db784aa98 WatchSource:0}: Error finding container ddee0bbc680d7b2fd81d2ace2fe52e9aae4a3e8e51e565be5f45477db784aa98: Status 404 returned error can't find the container with id ddee0bbc680d7b2fd81d2ace2fe52e9aae4a3e8e51e565be5f45477db784aa98
Oct 30 02:55:05.749320 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.749300    4377 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 02:55:05.757322 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.757299    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jtjnj"]
Oct 30 02:55:05.760292 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:05.760266    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod027c260d_eae4_4dda_834e_be9433d32534.slice/crio-8d91609255d43ae099c9b7d8018ba9d5669b414a25bc0c79b07b08e0c20a7e47 WatchSource:0}: Error finding container 8d91609255d43ae099c9b7d8018ba9d5669b414a25bc0c79b07b08e0c20a7e47: Status 404 returned error can't find the container with id 8d91609255d43ae099c9b7d8018ba9d5669b414a25bc0c79b07b08e0c20a7e47
Oct 30 02:55:05.763139 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:05.763118    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-h9c28"]
Oct 30 02:55:06.557650 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:06.557569    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9c28" event={"ID":"e2252025-152e-4341-9a75-285b38cc2436","Type":"ContainerStarted","Data":"517cd5d3dd6f50c3689f237328fde7d50a6744e3e16ebc1ce0023954f8b10eb5"}
Oct 30 02:55:06.559218 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:06.559180    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s258j" event={"ID":"fe685890-13f3-4526-921b-56a13e3dcea6","Type":"ContainerStarted","Data":"ddee0bbc680d7b2fd81d2ace2fe52e9aae4a3e8e51e565be5f45477db784aa98"}
Oct 30 02:55:06.561475 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:06.561449    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtjnj" event={"ID":"027c260d-eae4-4dda-834e-be9433d32534","Type":"ContainerStarted","Data":"516f767b0a878d39636f23c6c668d4fcbac6bbc007d2c61522ffc7bb37d57048"}
Oct 30 02:55:06.561633 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:06.561503    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtjnj" event={"ID":"027c260d-eae4-4dda-834e-be9433d32534","Type":"ContainerStarted","Data":"5e31058ae92b43bf30fc7c5f5321d26b6ee6af4215ea7644fa94c32186ebae13"}
Oct 30 02:55:06.561633 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:06.561550    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtjnj" event={"ID":"027c260d-eae4-4dda-834e-be9433d32534","Type":"ContainerStarted","Data":"8d91609255d43ae099c9b7d8018ba9d5669b414a25bc0c79b07b08e0c20a7e47"}
Oct 30 02:55:06.822180 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:06.822114    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:55:07.567210 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:07.567175    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s258j" event={"ID":"fe685890-13f3-4526-921b-56a13e3dcea6","Type":"ContainerStarted","Data":"4c221cc33a538422da386b16c42b6c81d516a58175f1695913fda180ca97caed"}
Oct 30 02:55:07.567565 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:07.567264    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s258j" event={"ID":"fe685890-13f3-4526-921b-56a13e3dcea6","Type":"ContainerStarted","Data":"d088a177d897ca3c7eb14dcbb6d94c23eceeae441a83c58a6ab3ba97dc5d7455"}
Oct 30 02:55:08.010504 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.010471    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-j9s8d"
Oct 30 02:55:08.573773 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.573738    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jtjnj" event={"ID":"027c260d-eae4-4dda-834e-be9433d32534","Type":"ContainerStarted","Data":"ccffc1c2aa745c03a26f4e8afb5dc735edf3519d73762ff7a140fd23f6123982"}
Oct 30 02:55:08.575514 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.575492    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9c28" event={"ID":"e2252025-152e-4341-9a75-285b38cc2436","Type":"ContainerStarted","Data":"a44fdb05374891021414a7e5c34959b678a03ca3d8168d4d5d4166aba3095d92"}
Oct 30 02:55:08.575583 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.575553    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-h9c28" event={"ID":"e2252025-152e-4341-9a75-285b38cc2436","Type":"ContainerStarted","Data":"61db21246f2c15162e509187e50a950517a87f284cac8d36f71ebad29f9f74b6"}
Oct 30 02:55:08.813256 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.813214    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-5gzvn"]
Oct 30 02:55:08.815004 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.814982    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"]
Oct 30 02:55:08.815124 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.815102    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:08.816977 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.816956    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"
Oct 30 02:55:08.819033 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.819012    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjvbq\" (UniqueName: \"kubernetes.io/projected/83bf340e-9d8a-4ec7-9308-4a449e878817-kube-api-access-wjvbq\") pod \"downloads-6dd789c8b7-5gzvn\" (UID: \"83bf340e-9d8a-4ec7-9308-4a449e878817\") " pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:08.819115 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.819043    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1b8014f7-f7ae-4d86-ab01-5d2878c532f7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2s9lx\" (UID: \"1b8014f7-f7ae-4d86-ab01-5d2878c532f7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"
Oct 30 02:55:08.819115 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.819082    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83bf340e-9d8a-4ec7-9308-4a449e878817-tmp\") pod \"downloads-6dd789c8b7-5gzvn\" (UID: \"83bf340e-9d8a-4ec7-9308-4a449e878817\") " pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:08.833362 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.833342    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-t4fvw\""
Oct 30 02:55:08.833362 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.833354    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ls6mp\""
Oct 30 02:55:08.833478 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.833437    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 02:55:08.833627 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.833614    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 02:55:08.833677 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.833628    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 02:55:08.845242 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.845220    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5gzvn"]
Oct 30 02:55:08.846692 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.846673    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"]
Oct 30 02:55:08.886887 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.886851    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56b49fb55b-sgq6q"]
Oct 30 02:55:08.890710 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.890683    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920495 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920465    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b593a669-e267-4333-b0e5-72f4a863aa56-trusted-ca\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920643 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920500    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-registry-tls\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920643 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920552    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b593a669-e267-4333-b0e5-72f4a863aa56-image-registry-private-configuration\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920643 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920620    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjvbq\" (UniqueName: \"kubernetes.io/projected/83bf340e-9d8a-4ec7-9308-4a449e878817-kube-api-access-wjvbq\") pod \"downloads-6dd789c8b7-5gzvn\" (UID: \"83bf340e-9d8a-4ec7-9308-4a449e878817\") " pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:08.920772 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920646    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1b8014f7-f7ae-4d86-ab01-5d2878c532f7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2s9lx\" (UID: \"1b8014f7-f7ae-4d86-ab01-5d2878c532f7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"
Oct 30 02:55:08.920772 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920671    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83bf340e-9d8a-4ec7-9308-4a449e878817-tmp\") pod \"downloads-6dd789c8b7-5gzvn\" (UID: \"83bf340e-9d8a-4ec7-9308-4a449e878817\") " pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:08.920772 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920696    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b593a669-e267-4333-b0e5-72f4a863aa56-registry-certificates\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920772 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920718    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-bound-sa-token\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920826    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b593a669-e267-4333-b0e5-72f4a863aa56-ca-trust-extracted\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920852    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqtnn\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-kube-api-access-jqtnn\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.920968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.920881    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b593a669-e267-4333-b0e5-72f4a863aa56-installation-pull-secrets\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:08.921125 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.921089    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83bf340e-9d8a-4ec7-9308-4a449e878817-tmp\") pod \"downloads-6dd789c8b7-5gzvn\" (UID: \"83bf340e-9d8a-4ec7-9308-4a449e878817\") " pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:08.922817 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.922801    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 02:55:08.923090 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.923071    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/1b8014f7-f7ae-4d86-ab01-5d2878c532f7-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2s9lx\" (UID: \"1b8014f7-f7ae-4d86-ab01-5d2878c532f7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"
Oct 30 02:55:08.928002 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.927910    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b49fb55b-sgq6q"]
Oct 30 02:55:08.932855 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.932832    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 02:55:08.932979 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.932834    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 02:55:08.932979 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.932927    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j9c5h\""
Oct 30 02:55:08.933078 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.933015    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 02:55:08.944532 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:08.944504    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjvbq\" (UniqueName: \"kubernetes.io/projected/83bf340e-9d8a-4ec7-9308-4a449e878817-kube-api-access-wjvbq\") pod \"downloads-6dd789c8b7-5gzvn\" (UID: \"83bf340e-9d8a-4ec7-9308-4a449e878817\") " pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:09.021328 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021292    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b593a669-e267-4333-b0e5-72f4a863aa56-installation-pull-secrets\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.021328 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021332    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b593a669-e267-4333-b0e5-72f4a863aa56-trusted-ca\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.021625 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021357    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-registry-tls\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.021625 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021385    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b593a669-e267-4333-b0e5-72f4a863aa56-image-registry-private-configuration\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.021625 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021421    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b593a669-e267-4333-b0e5-72f4a863aa56-registry-certificates\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.021625 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021442    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-bound-sa-token\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.021625 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021468    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b593a669-e267-4333-b0e5-72f4a863aa56-ca-trust-extracted\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.021625 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.021486    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqtnn\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-kube-api-access-jqtnn\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.022271 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.022231    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b593a669-e267-4333-b0e5-72f4a863aa56-ca-trust-extracted\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.022413 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.022391    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b593a669-e267-4333-b0e5-72f4a863aa56-registry-certificates\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.022692 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.022669    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b593a669-e267-4333-b0e5-72f4a863aa56-trusted-ca\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.024038 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.024018    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-registry-tls\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.024038 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.024024    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b593a669-e267-4333-b0e5-72f4a863aa56-image-registry-private-configuration\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.024151 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.024048    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b593a669-e267-4333-b0e5-72f4a863aa56-installation-pull-secrets\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.045093 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.045068    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-bound-sa-token\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.045215 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.045198    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqtnn\" (UniqueName: \"kubernetes.io/projected/b593a669-e267-4333-b0e5-72f4a863aa56-kube-api-access-jqtnn\") pod \"image-registry-56b49fb55b-sgq6q\" (UID: \"b593a669-e267-4333-b0e5-72f4a863aa56\") " pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.127865 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.127768    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:09.132500 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.132468    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"
Oct 30 02:55:09.199879 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.199831    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:09.436399 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.436307    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"]
Oct 30 02:55:09.436845 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.436821    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-5gzvn"]
Oct 30 02:55:09.442719 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:09.442688    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8014f7_f7ae_4d86_ab01_5d2878c532f7.slice/crio-4346649c7b5a28cebd970020ad1f40d1d55a9433c83219dddc1fbedf2e38c041 WatchSource:0}: Error finding container 4346649c7b5a28cebd970020ad1f40d1d55a9433c83219dddc1fbedf2e38c041: Status 404 returned error can't find the container with id 4346649c7b5a28cebd970020ad1f40d1d55a9433c83219dddc1fbedf2e38c041
Oct 30 02:55:09.443038 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:09.443016    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bf340e_9d8a_4ec7_9308_4a449e878817.slice/crio-52f7aa2cefd4b1e4a59408b571eea44b436c89cdd8b8dcd528f9fb4a86bbdc4a WatchSource:0}: Error finding container 52f7aa2cefd4b1e4a59408b571eea44b436c89cdd8b8dcd528f9fb4a86bbdc4a: Status 404 returned error can't find the container with id 52f7aa2cefd4b1e4a59408b571eea44b436c89cdd8b8dcd528f9fb4a86bbdc4a
Oct 30 02:55:09.500306 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.500277    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56b49fb55b-sgq6q"]
Oct 30 02:55:09.502960 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:09.502930    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb593a669_e267_4333_b0e5_72f4a863aa56.slice/crio-da1ba891e958e37f60294855d20f2bf4cd47c55d7e599a7256a2844bfcb54cf2 WatchSource:0}: Error finding container da1ba891e958e37f60294855d20f2bf4cd47c55d7e599a7256a2844bfcb54cf2: Status 404 returned error can't find the container with id da1ba891e958e37f60294855d20f2bf4cd47c55d7e599a7256a2844bfcb54cf2
Oct 30 02:55:09.581219 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.581187    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx" event={"ID":"1b8014f7-f7ae-4d86-ab01-5d2878c532f7","Type":"ContainerStarted","Data":"4346649c7b5a28cebd970020ad1f40d1d55a9433c83219dddc1fbedf2e38c041"}
Oct 30 02:55:09.582348 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.582321    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5gzvn" event={"ID":"83bf340e-9d8a-4ec7-9308-4a449e878817","Type":"ContainerStarted","Data":"52f7aa2cefd4b1e4a59408b571eea44b436c89cdd8b8dcd528f9fb4a86bbdc4a"}
Oct 30 02:55:09.583680 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.583654    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q" event={"ID":"b593a669-e267-4333-b0e5-72f4a863aa56","Type":"ContainerStarted","Data":"f61131cd25fec98bc82851f7a7c66145b391e124b8b3d384eec004ec67e242e0"}
Oct 30 02:55:09.583762 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.583700    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q" event={"ID":"b593a669-e267-4333-b0e5-72f4a863aa56","Type":"ContainerStarted","Data":"da1ba891e958e37f60294855d20f2bf4cd47c55d7e599a7256a2844bfcb54cf2"}
Oct 30 02:55:09.584128 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.584112    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:09.632942 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:09.632875    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q" podStartSLOduration=1.632855903 podStartE2EDuration="1.632855903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:09.631929963 +0000 UTC m=+9.224258770" watchObservedRunningTime="2026-10-30 02:55:09.632855903 +0000 UTC m=+9.225184711"
Oct 30 02:55:10.262019 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:10.261986    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:55:10.586344 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:10.586317    4377 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:11.348195 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:11.348168    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:11.348428 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:11.348413    4377 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:11.591295 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:11.591248    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx" event={"ID":"1b8014f7-f7ae-4d86-ab01-5d2878c532f7","Type":"ContainerStarted","Data":"2a97663a7f3895a752ee090eb4cfd0300ee26fbf198bf68ca7fb4115ea4bb6d7"}
Oct 30 02:55:11.639091 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:11.638992    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx" podStartSLOduration=2.562734786 podStartE2EDuration="3.638975267s" totalImagesPullingTime="1.076240481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:11.637977047 +0000 UTC m=+11.230305853" watchObservedRunningTime="2026-10-30 02:55:11.638975267 +0000 UTC m=+11.231304074"
Oct 30 02:55:12.594254 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:12.594225    4377 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:16.094900 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:16.094861    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"
Oct 30 02:55:16.095351 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:16.094988    4377 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 02:55:16.100172 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:16.100149    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2s9lx"
Oct 30 02:55:17.836607 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:17.836566    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:18.565292 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.565248    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b878c694d-5jpjh"]
Oct 30 02:55:18.567436 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.567414    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.582841 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.582816    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 02:55:18.585825 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.585802    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4sq8z\""
Oct 30 02:55:18.587136 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.587108    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-config\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.587263 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.587151    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-oauth-serving-cert\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.587263 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.587182    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsm7h\" (UniqueName: \"kubernetes.io/projected/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-kube-api-access-zsm7h\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.587362 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.587275    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-serving-cert\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.587362 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.587341    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-oauth-config\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.587444 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.587382    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-tmp\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.587444 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.587407    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-service-ca\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.595785 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.595758    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 02:55:18.596928 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.596858    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b878c694d-5jpjh"]
Oct 30 02:55:18.609072 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.609046    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 02:55:18.609072 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.609047    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 02:55:18.686471 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.686431    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 02:55:18.688686 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.688659    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsm7h\" (UniqueName: \"kubernetes.io/projected/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-kube-api-access-zsm7h\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.688825 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.688702    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-serving-cert\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.688825 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.688743    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-oauth-config\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.688825 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.688784    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-tmp\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.688825 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.688820    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-service-ca\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.689046 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.688871    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-config\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.689046 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.688898    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-oauth-serving-cert\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.689294 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.689243    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-tmp\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.689673 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.689626    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-oauth-serving-cert\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.689803 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.689777    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-service-ca\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.689857 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.689786    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-config\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.692051 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.692026    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-oauth-config\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.697170 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.697143    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-serving-cert\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.712301 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.712280    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsm7h\" (UniqueName: \"kubernetes.io/projected/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-kube-api-access-zsm7h\") pod \"console-6b878c694d-5jpjh\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") " pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:18.879381 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:18.879292    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:19.186837 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:19.186744    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b878c694d-5jpjh"]
Oct 30 02:55:19.189985 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:19.189955    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe07cb35_0598_4e51_8bba_b6dbdaa6441b.slice/crio-f1bb9490247c2d003e9fea5c834052d82acbbc80ab0e1ff5b89639ec42ee9b09 WatchSource:0}: Error finding container f1bb9490247c2d003e9fea5c834052d82acbbc80ab0e1ff5b89639ec42ee9b09: Status 404 returned error can't find the container with id f1bb9490247c2d003e9fea5c834052d82acbbc80ab0e1ff5b89639ec42ee9b09
Oct 30 02:55:19.589655 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:19.589623    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-h9c28"
Oct 30 02:55:19.618481 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:19.618445    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b878c694d-5jpjh" event={"ID":"fe07cb35-0598-4e51-8bba-b6dbdaa6441b","Type":"ContainerStarted","Data":"f1bb9490247c2d003e9fea5c834052d82acbbc80ab0e1ff5b89639ec42ee9b09"}
Oct 30 02:55:22.022628 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.022594    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"]
Oct 30 02:55:22.025789 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.025765    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.035373 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.035352    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5wnpc"]
Oct 30 02:55:22.038476 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.038454    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.046870 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.046844    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 02:55:22.047371 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.047339    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 02:55:22.047471 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.047407    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 02:55:22.060141 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.060116    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 02:55:22.061445 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.061360    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"]
Oct 30 02:55:22.091826 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.091802    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 02:55:22.091963 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.091875    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 02:55:22.092060 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.091984    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4s5rs\""
Oct 30 02:55:22.092060 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.091994    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 02:55:22.092060 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.092015    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gmwmp\""
Oct 30 02:55:22.092060 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.092012    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 02:55:22.115544 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115503    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-sys\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.115678 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115554    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgv9k\" (UniqueName: \"kubernetes.io/projected/eafa9117-aa83-435f-bdc4-9652062d448e-kube-api-access-jgv9k\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.115678 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115668    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-textfile\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.115796 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115711    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.115796 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115731    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.115796 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115751    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcjqs\" (UniqueName: \"kubernetes.io/projected/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-api-access-kcjqs\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.115796 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115780    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.115999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115819    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.115999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115845    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-wtmp\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.115999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115875    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.115999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115894    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.115999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115917    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-tls\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.115999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115942    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.115999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.115974    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-root\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.116340 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.116001    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eafa9117-aa83-435f-bdc4-9652062d448e-metrics-client-ca\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.146424 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.146374    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 02:55:22.216871 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.216834    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-sys\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.216871 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.216878    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgv9k\" (UniqueName: \"kubernetes.io/projected/eafa9117-aa83-435f-bdc4-9652062d448e-kube-api-access-jgv9k\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.216915    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-textfile\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.216947    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.216952    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-sys\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.216968    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.216997    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcjqs\" (UniqueName: \"kubernetes.io/projected/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-api-access-kcjqs\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217028    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217066    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217088    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-wtmp\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217118    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217556 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217145    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.217556 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217178    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-tls\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217556 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217201    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217556 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217229    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-root\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217556 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217261    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eafa9117-aa83-435f-bdc4-9652062d448e-metrics-client-ca\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217556 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217292    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-wtmp\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217556 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217328    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-textfile\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217826 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217600    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eafa9117-aa83-435f-bdc4-9652062d448e-root\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.217826 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217763    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.217826 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217816    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eafa9117-aa83-435f-bdc4-9652062d448e-metrics-client-ca\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.218004 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.217979    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-accelerators-collector-config\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.218109 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.218089    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.218170 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.218113    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.220474 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.220302    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.220474 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.220436    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-tls\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.220642 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.220513    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.222968 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.222946    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eafa9117-aa83-435f-bdc4-9652062d448e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.240326 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.240300    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgv9k\" (UniqueName: \"kubernetes.io/projected/eafa9117-aa83-435f-bdc4-9652062d448e-kube-api-access-jgv9k\") pod \"node-exporter-5wnpc\" (UID: \"eafa9117-aa83-435f-bdc4-9652062d448e\") " pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.240629 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.240610    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcjqs\" (UniqueName: \"kubernetes.io/projected/9dbb7bb1-9c6c-4eef-824a-84a3adccbe35-kube-api-access-kcjqs\") pod \"kube-state-metrics-5dc8d7d89b-z5qvn\" (UID: \"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.344238 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.344205    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"
Oct 30 02:55:22.356578 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.356553    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5wnpc"
Oct 30 02:55:22.368148 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:22.368117    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafa9117_aa83_435f_bdc4_9652062d448e.slice/crio-69f10ba073505b29d5eb372b964f2001c830e6ae116155784d704c353eb30c32 WatchSource:0}: Error finding container 69f10ba073505b29d5eb372b964f2001c830e6ae116155784d704c353eb30c32: Status 404 returned error can't find the container with id 69f10ba073505b29d5eb372b964f2001c830e6ae116155784d704c353eb30c32
Oct 30 02:55:22.633260 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.633178    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b878c694d-5jpjh" event={"ID":"fe07cb35-0598-4e51-8bba-b6dbdaa6441b","Type":"ContainerStarted","Data":"c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9"}
Oct 30 02:55:22.634607 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.634578    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wnpc" event={"ID":"eafa9117-aa83-435f-bdc4-9652062d448e","Type":"ContainerStarted","Data":"69f10ba073505b29d5eb372b964f2001c830e6ae116155784d704c353eb30c32"}
Oct 30 02:55:22.663949 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.663918    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn"]
Oct 30 02:55:22.677116 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:22.677075    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6b878c694d-5jpjh" podStartSLOduration=1.828657265 podStartE2EDuration="4.677061248s" totalImagesPullingTime="2.848403983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:22.676515639 +0000 UTC m=+22.268844446" watchObservedRunningTime="2026-10-30 02:55:22.677061248 +0000 UTC m=+22.269390054"
Oct 30 02:55:23.641229 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:23.641201    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wnpc" event={"ID":"eafa9117-aa83-435f-bdc4-9652062d448e","Type":"ContainerStarted","Data":"18b8b43baf8880f72616f36828313896b1466e44ca43dcaee3f93e4610210ae5"}
Oct 30 02:55:23.643022 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:23.642984    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn" event={"ID":"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35","Type":"ContainerStarted","Data":"eba34f457839ef243437464915c1bdf22eeda3e522c2159c9459bf9e34d32719"}
Oct 30 02:55:24.651493 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:24.651458    4377 generic.go:356] "Generic (PLEG): container finished" podID="eafa9117-aa83-435f-bdc4-9652062d448e" containerID="18b8b43baf8880f72616f36828313896b1466e44ca43dcaee3f93e4610210ae5" exitCode=0
Oct 30 02:55:24.651954 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:24.651589    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wnpc" event={"ID":"eafa9117-aa83-435f-bdc4-9652062d448e","Type":"ContainerDied","Data":"18b8b43baf8880f72616f36828313896b1466e44ca43dcaee3f93e4610210ae5"}
Oct 30 02:55:24.651954 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:24.651654    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wnpc" event={"ID":"eafa9117-aa83-435f-bdc4-9652062d448e","Type":"ContainerStarted","Data":"ff633f2b1d1e8961218705cc8fdaa59aa25e562a5ebc8d841c8f080adf394527"}
Oct 30 02:55:24.651954 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:24.651763    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5wnpc" event={"ID":"eafa9117-aa83-435f-bdc4-9652062d448e","Type":"ContainerStarted","Data":"860c2e53f6c09d272ebed7355aec228a22af19edb8f509cc5ae80978051346ab"}
Oct 30 02:55:24.656028 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:24.655755    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn" event={"ID":"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35","Type":"ContainerStarted","Data":"ad194d0b2671848a9077c86fbd50d3997aa6f388cc03ad9e72893f7e8e698227"}
Oct 30 02:55:24.656028 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:24.655825    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn" event={"ID":"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35","Type":"ContainerStarted","Data":"b1ae57c4f0b707e10a7b4914e6f2511e5c86a8ee854c461ae350d785adaf4b28"}
Oct 30 02:55:24.724263 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:24.724177    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5wnpc" podStartSLOduration=2.561001057 podStartE2EDuration="3.724161869s" totalImagesPullingTime="1.078144636s" totalInitContainerRuntime="85.016176ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:24.712774108 +0000 UTC m=+24.305102914" watchObservedRunningTime="2026-10-30 02:55:24.724161869 +0000 UTC m=+24.316490680"
Oct 30 02:55:25.661516 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:25.661477    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn" event={"ID":"9dbb7bb1-9c6c-4eef-824a-84a3adccbe35","Type":"ContainerStarted","Data":"38aaec00f9912adc072cc3abc939b0e51fa6a920a76dd3402d9c2f8dae862795"}
Oct 30 02:55:26.317823 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.317759    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-z5qvn" podStartSLOduration=3.6305388990000003 podStartE2EDuration="5.317737419s" totalImagesPullingTime="1.68719852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:25.733806143 +0000 UTC m=+25.326134949" watchObservedRunningTime="2026-10-30 02:55:26.317737419 +0000 UTC m=+25.910066225"
Oct 30 02:55:26.318617 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.318583    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dc7d9f9cf-f4qj5"]
Oct 30 02:55:26.321503 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.321477    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.354732 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.354694    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-oauth-config\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.354913 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.354762    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nt5n\" (UniqueName: \"kubernetes.io/projected/f949d6f0-2ccf-4662-9dd9-9073d19b8704-kube-api-access-5nt5n\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.354913 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.354867    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-serving-cert\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.355042 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.354926    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-config\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.355098 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.355041    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f949d6f0-2ccf-4662-9dd9-9073d19b8704-tmp\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.355098 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.355090    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-service-ca\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.355197 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.355122    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-trusted-ca-bundle\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.355252 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.355210    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-oauth-serving-cert\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.377016 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.376985    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 02:55:26.378212 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.378187    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc7d9f9cf-f4qj5"]
Oct 30 02:55:26.456677 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.456638    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-oauth-serving-cert\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457405 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457378    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-oauth-serving-cert\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457513 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457477    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-oauth-config\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457610 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457538    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nt5n\" (UniqueName: \"kubernetes.io/projected/f949d6f0-2ccf-4662-9dd9-9073d19b8704-kube-api-access-5nt5n\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457610 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457585    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-serving-cert\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457716 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457612    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-config\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457716 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457658    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f949d6f0-2ccf-4662-9dd9-9073d19b8704-tmp\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457834 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457804    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-service-ca\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457894 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457863    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-trusted-ca-bundle\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.457991 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.457971    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f949d6f0-2ccf-4662-9dd9-9073d19b8704-tmp\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.458370 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.458331    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-config\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.458716 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.458692    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-service-ca\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.458812 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.458782    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-trusted-ca-bundle\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.461058 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.461036    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-serving-cert\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.461147 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.461083    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-oauth-config\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.465753 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.465719    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-677996c9cc-w657x"]
Oct 30 02:55:26.471023 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.471000    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.483967 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.483944    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nt5n\" (UniqueName: \"kubernetes.io/projected/f949d6f0-2ccf-4662-9dd9-9073d19b8704-kube-api-access-5nt5n\") pod \"console-5dc7d9f9cf-f4qj5\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") " pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.492733 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.492646    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-677996c9cc-w657x"]
Oct 30 02:55:26.493224 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.493202    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 02:55:26.493224 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.493214    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 02:55:26.496374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.494902    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 02:55:26.496374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.495181    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9kggiuprbqrns\""
Oct 30 02:55:26.496374 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.495197    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-rk26h\""
Oct 30 02:55:26.498223 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.498202    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 02:55:26.559124 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.559086    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-client-ca-bundle\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.559313 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.559139    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-audit-log\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.559313 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.559227    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-secret-metrics-server-tls\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.559313 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.559287    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.559471 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.559345    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ss8h\" (UniqueName: \"kubernetes.io/projected/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-kube-api-access-5ss8h\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.559471 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.559377    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-metrics-server-audit-profiles\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.559471 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.559431    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-secret-metrics-server-client-certs\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.634327 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.634246    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:26.660457 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.660409    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.660604 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.660473    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ss8h\" (UniqueName: \"kubernetes.io/projected/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-kube-api-access-5ss8h\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.660604 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.660506    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-metrics-server-audit-profiles\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.660604 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.660576    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-secret-metrics-server-client-certs\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.660777 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.660640    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-client-ca-bundle\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.660777 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.660685    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-audit-log\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.660777 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.660711    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-secret-metrics-server-tls\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.661907 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.661240    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.662297 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.662075    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-metrics-server-audit-profiles\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.662420 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.662396    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-audit-log\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.663846 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.663822    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-secret-metrics-server-tls\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.664434 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.664347    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-secret-metrics-server-client-certs\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.665255 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.665228    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-client-ca-bundle\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.670160 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.670129    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-5gzvn" event={"ID":"83bf340e-9d8a-4ec7-9308-4a449e878817","Type":"ContainerStarted","Data":"576f6ef5d53b4299c989448a1da70747e7a7806ce3799266ae9d6704afe002c7"}
Oct 30 02:55:26.670554 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.670515    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:26.683638 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.683612    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-5gzvn"
Oct 30 02:55:26.686080 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.686053    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ss8h\" (UniqueName: \"kubernetes.io/projected/f05cafde-12ba-4845-9ad7-8f91be4e3ba8-kube-api-access-5ss8h\") pod \"metrics-server-677996c9cc-w657x\" (UID: \"f05cafde-12ba-4845-9ad7-8f91be4e3ba8\") " pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.716657 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.716601    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-5gzvn" podStartSLOduration=2.454539553 podStartE2EDuration="18.716583058s" totalImagesPullingTime="16.262043505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:26.715722232 +0000 UTC m=+26.308051057" watchObservedRunningTime="2026-10-30 02:55:26.716583058 +0000 UTC m=+26.308911865"
Oct 30 02:55:26.786095 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.786054    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:26.965126 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:26.965090    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dc7d9f9cf-f4qj5"]
Oct 30 02:55:26.968836 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:26.968803    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf949d6f0_2ccf_4662_9dd9_9073d19b8704.slice/crio-4afc8df622fd94fc117d625ab4af51c2738f4150442617a429eabe13bded870a WatchSource:0}: Error finding container 4afc8df622fd94fc117d625ab4af51c2738f4150442617a429eabe13bded870a: Status 404 returned error can't find the container with id 4afc8df622fd94fc117d625ab4af51c2738f4150442617a429eabe13bded870a
Oct 30 02:55:27.130196 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:27.130163    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-677996c9cc-w657x"]
Oct 30 02:55:27.134282 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:27.134245    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf05cafde_12ba_4845_9ad7_8f91be4e3ba8.slice/crio-8befe7e4384e738d2da4396350bc55684e3d5af02032c95f12b9d6501b22ea65 WatchSource:0}: Error finding container 8befe7e4384e738d2da4396350bc55684e3d5af02032c95f12b9d6501b22ea65: Status 404 returned error can't find the container with id 8befe7e4384e738d2da4396350bc55684e3d5af02032c95f12b9d6501b22ea65
Oct 30 02:55:27.685945 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:27.685824    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-677996c9cc-w657x" event={"ID":"f05cafde-12ba-4845-9ad7-8f91be4e3ba8","Type":"ContainerStarted","Data":"8befe7e4384e738d2da4396350bc55684e3d5af02032c95f12b9d6501b22ea65"}
Oct 30 02:55:27.688240 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:27.688194    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc7d9f9cf-f4qj5" event={"ID":"f949d6f0-2ccf-4662-9dd9-9073d19b8704","Type":"ContainerStarted","Data":"87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6"}
Oct 30 02:55:27.688450 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:27.688428    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc7d9f9cf-f4qj5" event={"ID":"f949d6f0-2ccf-4662-9dd9-9073d19b8704","Type":"ContainerStarted","Data":"4afc8df622fd94fc117d625ab4af51c2738f4150442617a429eabe13bded870a"}
Oct 30 02:55:27.843276 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:27.843220    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56b49fb55b-sgq6q"
Oct 30 02:55:27.896440 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:27.895951    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5dc7d9f9cf-f4qj5" podStartSLOduration=1.895930247 podStartE2EDuration="1.895930247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:27.734708581 +0000 UTC m=+27.327037388" watchObservedRunningTime="2026-10-30 02:55:27.895930247 +0000 UTC m=+27.488259054"
Oct 30 02:55:28.880010 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:28.879967    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:28.880609 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:28.880050    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:28.885378 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:28.885355    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:29.697988 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:29.697948    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-677996c9cc-w657x" event={"ID":"f05cafde-12ba-4845-9ad7-8f91be4e3ba8","Type":"ContainerStarted","Data":"4a7e1eab3f3d50591c21a201915af2214c5c7deb7fbec69ce9bf3aa22880778b"}
Oct 30 02:55:29.702200 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:29.702172    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:55:29.789491 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:29.789436    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-677996c9cc-w657x" podStartSLOduration=1.8458803430000001 podStartE2EDuration="3.789416215s" totalImagesPullingTime="1.943535872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:29.74382534 +0000 UTC m=+29.336154148" watchObservedRunningTime="2026-10-30 02:55:29.789416215 +0000 UTC m=+29.381745022"
Oct 30 02:55:30.759143 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:30.759111    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fvzlt"
Oct 30 02:55:32.167821 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.167781    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dc7d9f9cf-f4qj5"]
Oct 30 02:55:32.203006 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.202974    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-666fdc54bd-kb7nv"]
Oct 30 02:55:32.231421 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.231393    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666fdc54bd-kb7nv"]
Oct 30 02:55:32.231608 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.231543    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.306941 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.306904    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-console-config\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.306941 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.306942    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-oauth-serving-cert\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.307158 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.306972    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-oauth-config\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.307158 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.307079    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knwlz\" (UniqueName: \"kubernetes.io/projected/bede845b-baec-4a7c-9fd3-eb6025701107-kube-api-access-knwlz\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.307252 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.307170    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bede845b-baec-4a7c-9fd3-eb6025701107-tmp\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.307252 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.307237    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-serving-cert\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.307341 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.307257    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-trusted-ca-bundle\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.307341 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.307278    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-service-ca\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.407885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.407843    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-service-ca\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.407885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.407893    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-console-config\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408139 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.407912    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-oauth-serving-cert\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408139 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.407943    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-oauth-config\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408139 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.407967    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knwlz\" (UniqueName: \"kubernetes.io/projected/bede845b-baec-4a7c-9fd3-eb6025701107-kube-api-access-knwlz\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408139 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.408022    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bede845b-baec-4a7c-9fd3-eb6025701107-tmp\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408139 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.408091    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-serving-cert\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408139 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.408122    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-trusted-ca-bundle\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408726 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.408687    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-console-config\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408843 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.408788    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-service-ca\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.408989 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.408969    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bede845b-baec-4a7c-9fd3-eb6025701107-tmp\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.409754 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.409726    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-oauth-serving-cert\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.409918 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.409896    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-trusted-ca-bundle\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.411167 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.411122    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-oauth-config\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.411594 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.411572    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-serving-cert\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.433349 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.433284    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knwlz\" (UniqueName: \"kubernetes.io/projected/bede845b-baec-4a7c-9fd3-eb6025701107-kube-api-access-knwlz\") pod \"console-666fdc54bd-kb7nv\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") " pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.557929 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.557889    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:32.902201 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:32.902165    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-666fdc54bd-kb7nv"]
Oct 30 02:55:32.905580 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:55:32.905549    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbede845b_baec_4a7c_9fd3_eb6025701107.slice/crio-1760a89ed47c93f8f99cc047a1849fdc0a06d9a6dd2341006a6ebb26cac6c01f WatchSource:0}: Error finding container 1760a89ed47c93f8f99cc047a1849fdc0a06d9a6dd2341006a6ebb26cac6c01f: Status 404 returned error can't find the container with id 1760a89ed47c93f8f99cc047a1849fdc0a06d9a6dd2341006a6ebb26cac6c01f
Oct 30 02:55:33.715457 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:33.715417    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fdc54bd-kb7nv" event={"ID":"bede845b-baec-4a7c-9fd3-eb6025701107","Type":"ContainerStarted","Data":"28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4"}
Oct 30 02:55:33.715921 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:33.715479    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fdc54bd-kb7nv" event={"ID":"bede845b-baec-4a7c-9fd3-eb6025701107","Type":"ContainerStarted","Data":"1760a89ed47c93f8f99cc047a1849fdc0a06d9a6dd2341006a6ebb26cac6c01f"}
Oct 30 02:55:33.777314 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:33.777253    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-666fdc54bd-kb7nv" podStartSLOduration=1.777232256 podStartE2EDuration="1.777232256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:33.763877145 +0000 UTC m=+33.356205951" watchObservedRunningTime="2026-10-30 02:55:33.777232256 +0000 UTC m=+33.369561063"
Oct 30 02:55:36.634789 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:36.634750    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:36.825105 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:36.825072    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mrnst"
Oct 30 02:55:40.257838 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:40.257800    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:55:42.558010 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:42.557971    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:42.558721 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:42.558029    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:42.562739 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:42.562715    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:42.747311 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:42.747277    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:55:42.825916 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:42.825833    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b878c694d-5jpjh"]
Oct 30 02:55:46.787137 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:46.787060    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:46.787137 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:46.787120    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:55:57.189976 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.189907    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5dc7d9f9cf-f4qj5" podUID="f949d6f0-2ccf-4662-9dd9-9073d19b8704" containerName="console" containerID="cri-o://87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6" gracePeriod=15
Oct 30 02:55:57.452299 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.452235    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:57.622462 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622415    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-oauth-config\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622653 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622494    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-serving-cert\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622653 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622539    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f949d6f0-2ccf-4662-9dd9-9073d19b8704-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f949d6f0-2ccf-4662-9dd9-9073d19b8704-tmp\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622653 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622569    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-service-ca\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-service-ca\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622653 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622609    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-oauth-serving-cert\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622874 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622670    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f949d6f0-2ccf-4662-9dd9-9073d19b8704-kube-api-access-5nt5n\" (UniqueName: \"kubernetes.io/projected/f949d6f0-2ccf-4662-9dd9-9073d19b8704-kube-api-access-5nt5n\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622874 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622701    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-trusted-ca-bundle\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622874 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622743    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-config\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-config\") pod \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\" (UID: \"f949d6f0-2ccf-4662-9dd9-9073d19b8704\") "
Oct 30 02:55:57.622874 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.622821    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f949d6f0-2ccf-4662-9dd9-9073d19b8704-tmp" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:55:57.623082 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.623028    4377 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f949d6f0-2ccf-4662-9dd9-9073d19b8704-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.623082 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.623026    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-service-ca" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:57.623082 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.623030    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-oauth-serving-cert" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:57.623337 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.623298    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-trusted-ca-bundle" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:57.623486 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.623469    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-config" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:55:57.624788 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.624763    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-oauth-config" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:57.625050 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.625030    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-serving-cert" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:55:57.625104 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.625031    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f949d6f0-2ccf-4662-9dd9-9073d19b8704-kube-api-access-5nt5n" pod "f949d6f0-2ccf-4662-9dd9-9073d19b8704" (UID: "f949d6f0-2ccf-4662-9dd9-9073d19b8704"). InnerVolumeSpecName "kube-api-access-5nt5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:55:57.723509 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.723429    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nt5n\" (UniqueName: \"kubernetes.io/projected/f949d6f0-2ccf-4662-9dd9-9073d19b8704-kube-api-access-5nt5n\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.723509 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.723461    4377 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.723509 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.723474    4377 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.723509 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.723486    4377 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.723509 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.723498    4377 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f949d6f0-2ccf-4662-9dd9-9073d19b8704-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.723509 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.723510    4377 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.723830 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.723544    4377 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f949d6f0-2ccf-4662-9dd9-9073d19b8704-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:55:57.792414 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.792380    4377 generic.go:356] "Generic (PLEG): container finished" podID="f949d6f0-2ccf-4662-9dd9-9073d19b8704" containerID="87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6" exitCode=0
Oct 30 02:55:57.792612 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.792435    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc7d9f9cf-f4qj5" event={"ID":"f949d6f0-2ccf-4662-9dd9-9073d19b8704","Type":"ContainerDied","Data":"87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6"}
Oct 30 02:55:57.792612 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.792442    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dc7d9f9cf-f4qj5"
Oct 30 02:55:57.792612 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.792484    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dc7d9f9cf-f4qj5" event={"ID":"f949d6f0-2ccf-4662-9dd9-9073d19b8704","Type":"ContainerDied","Data":"4afc8df622fd94fc117d625ab4af51c2738f4150442617a429eabe13bded870a"}
Oct 30 02:55:57.792612 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.792512    4377 scope.go:122] "RemoveContainer" containerID="87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6"
Oct 30 02:55:57.801265 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.801241    4377 scope.go:122] "RemoveContainer" containerID="87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6"
Oct 30 02:55:57.801505 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:55:57.801483    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6\": container with ID starting with 87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6 not found: ID does not exist" containerID="87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6"
Oct 30 02:55:57.801579 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.801514    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6"} err="failed to get container status \"87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6\": rpc error: code = NotFound desc = could not find container \"87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6\": container with ID starting with 87d245eb51c6a200df06f78d5a93a91d08b3032a7015d6d320441310223bc4f6 not found: ID does not exist"
Oct 30 02:55:57.844844 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.844823    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dc7d9f9cf-f4qj5"]
Oct 30 02:55:57.848776 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:57.848758    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dc7d9f9cf-f4qj5"]
Oct 30 02:55:58.516729 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:55:58.515013    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f949d6f0-2ccf-4662-9dd9-9073d19b8704" path="/var/lib/kubelet/pods/f949d6f0-2ccf-4662-9dd9-9073d19b8704/volumes"
Oct 30 02:56:06.791406 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:06.791377    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:56:06.795203 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:06.795178    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-677996c9cc-w657x"
Oct 30 02:56:07.848135 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:07.848077    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6b878c694d-5jpjh" podUID="fe07cb35-0598-4e51-8bba-b6dbdaa6441b" containerName="console" containerID="cri-o://c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9" gracePeriod=15
Oct 30 02:56:08.111360 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.111297    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:56:08.194240 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194213    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-service-ca\") pod \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") "
Oct 30 02:56:08.194398 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194247    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-oauth-serving-cert\") pod \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") "
Oct 30 02:56:08.194398 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194282    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-kube-api-access-zsm7h\" (UniqueName: \"kubernetes.io/projected/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-kube-api-access-zsm7h\") pod \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") "
Oct 30 02:56:08.194398 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194316    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-tmp\") pod \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") "
Oct 30 02:56:08.194398 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194338    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-serving-cert\") pod \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") "
Oct 30 02:56:08.194398 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194372    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-config\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-config\") pod \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") "
Oct 30 02:56:08.194683 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194401    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-oauth-config\") pod \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\" (UID: \"fe07cb35-0598-4e51-8bba-b6dbdaa6441b\") "
Oct 30 02:56:08.194683 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194660    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-tmp" pod "fe07cb35-0598-4e51-8bba-b6dbdaa6441b" (UID: "fe07cb35-0598-4e51-8bba-b6dbdaa6441b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:56:08.194747 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194694    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-service-ca" pod "fe07cb35-0598-4e51-8bba-b6dbdaa6441b" (UID: "fe07cb35-0598-4e51-8bba-b6dbdaa6441b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:08.194747 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.194729    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-oauth-serving-cert" pod "fe07cb35-0598-4e51-8bba-b6dbdaa6441b" (UID: "fe07cb35-0598-4e51-8bba-b6dbdaa6441b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:08.195063 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.195038    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-config" pod "fe07cb35-0598-4e51-8bba-b6dbdaa6441b" (UID: "fe07cb35-0598-4e51-8bba-b6dbdaa6441b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:08.196495 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.196469    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-oauth-config" pod "fe07cb35-0598-4e51-8bba-b6dbdaa6441b" (UID: "fe07cb35-0598-4e51-8bba-b6dbdaa6441b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:08.196619 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.196558    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-serving-cert" pod "fe07cb35-0598-4e51-8bba-b6dbdaa6441b" (UID: "fe07cb35-0598-4e51-8bba-b6dbdaa6441b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:08.196619 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.196562    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-kube-api-access-zsm7h" pod "fe07cb35-0598-4e51-8bba-b6dbdaa6441b" (UID: "fe07cb35-0598-4e51-8bba-b6dbdaa6441b"). InnerVolumeSpecName "kube-api-access-zsm7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:56:08.295515 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.295487    4377 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:08.295515 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.295510    4377 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:08.295515 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.295534    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsm7h\" (UniqueName: \"kubernetes.io/projected/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-kube-api-access-zsm7h\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:08.295713 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.295544    4377 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:08.295713 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.295553    4377 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:08.295713 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.295562    4377 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:08.295713 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.295570    4377 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe07cb35-0598-4e51-8bba-b6dbdaa6441b-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:08.829686 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.829652    4377 generic.go:356] "Generic (PLEG): container finished" podID="fe07cb35-0598-4e51-8bba-b6dbdaa6441b" containerID="c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9" exitCode=0
Oct 30 02:56:08.829860 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.829715    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b878c694d-5jpjh"
Oct 30 02:56:08.829860 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.829739    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b878c694d-5jpjh" event={"ID":"fe07cb35-0598-4e51-8bba-b6dbdaa6441b","Type":"ContainerDied","Data":"c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9"}
Oct 30 02:56:08.829860 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.829804    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b878c694d-5jpjh" event={"ID":"fe07cb35-0598-4e51-8bba-b6dbdaa6441b","Type":"ContainerDied","Data":"f1bb9490247c2d003e9fea5c834052d82acbbc80ab0e1ff5b89639ec42ee9b09"}
Oct 30 02:56:08.829860 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.829826    4377 scope.go:122] "RemoveContainer" containerID="c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9"
Oct 30 02:56:08.841901 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.841884    4377 scope.go:122] "RemoveContainer" containerID="c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9"
Oct 30 02:56:08.842176 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:56:08.842153    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9\": container with ID starting with c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9 not found: ID does not exist" containerID="c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9"
Oct 30 02:56:08.842244 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.842184    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9"} err="failed to get container status \"c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9\": rpc error: code = NotFound desc = could not find container \"c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9\": container with ID starting with c460f16720566cbe3782b423c9f44a9d1f6fe7ae855389564adcfaba19d704b9 not found: ID does not exist"
Oct 30 02:56:08.882515 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.882490    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b878c694d-5jpjh"]
Oct 30 02:56:08.894958 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:08.894936    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b878c694d-5jpjh"]
Oct 30 02:56:10.258668 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:56:10.258634    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:56:10.513366 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:10.513289    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe07cb35-0598-4e51-8bba-b6dbdaa6441b" path="/var/lib/kubelet/pods/fe07cb35-0598-4e51-8bba-b6dbdaa6441b/volumes"
Oct 30 02:56:17.470454 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.470415    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"]
Oct 30 02:56:17.470935 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.470689    4377 cpu_manager.go:414] "containerMap: removing container" podUID="f949d6f0-2ccf-4662-9dd9-9073d19b8704" containerName="console"
Oct 30 02:56:17.470935 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.470701    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f949d6f0-2ccf-4662-9dd9-9073d19b8704" containerName="console"
Oct 30 02:56:17.470935 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.470718    4377 cpu_manager.go:414] "containerMap: removing container" podUID="fe07cb35-0598-4e51-8bba-b6dbdaa6441b" containerName="console"
Oct 30 02:56:17.470935 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.470723    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe07cb35-0598-4e51-8bba-b6dbdaa6441b" containerName="console"
Oct 30 02:56:17.470935 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.470767    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe07cb35-0598-4e51-8bba-b6dbdaa6441b" containerName="console"
Oct 30 02:56:17.470935 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.470777    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="f949d6f0-2ccf-4662-9dd9-9073d19b8704" containerName="console"
Oct 30 02:56:17.472723 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.472707    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.504975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.504948    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"]
Oct 30 02:56:17.531240 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.531209    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 02:56:17.531240 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.531231    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 02:56:17.533778 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.533758    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 02:56:17.533778 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.533771    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 02:56:17.533871 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.533847    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6wvz9\""
Oct 30 02:56:17.561326 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561299    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9grl8\" (UniqueName: \"kubernetes.io/projected/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-kube-api-access-9grl8\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.561428 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561332    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-secret-telemeter-client\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.561428 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561353    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-federate-client-tls\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.561428 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561371    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-metrics-client-ca\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.561543 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561506    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-telemeter-client-tls\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.561582 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561552    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.561582 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561576    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.561645 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.561605    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-serving-certs-ca-bundle\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.567025 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.567006    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 02:56:17.580180 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.580150    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 02:56:17.662769 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.662720    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-federate-client-tls\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.662769 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.662770    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-metrics-client-ca\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663031 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.662893    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-telemeter-client-tls\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663031 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.662937    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663031 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.662969    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663031 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.663020    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-serving-certs-ca-bundle\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663231 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.663059    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9grl8\" (UniqueName: \"kubernetes.io/projected/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-kube-api-access-9grl8\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663231 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.663099    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-secret-telemeter-client\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663623 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.663600    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-metrics-client-ca\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.663793 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.663760    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-serving-certs-ca-bundle\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.664007 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.663982    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.665492 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.665468    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-federate-client-tls\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.665600 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.665504    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.665770 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.665753    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-secret-telemeter-client\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.665810 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.665791    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-telemeter-client-tls\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.688609 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.688582    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9grl8\" (UniqueName: \"kubernetes.io/projected/3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4-kube-api-access-9grl8\") pod \"telemeter-client-599ff67b9b-k8vwn\" (UID: \"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4\") " pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:17.789485 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:17.789453    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"
Oct 30 02:56:18.135670 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:18.135639    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-599ff67b9b-k8vwn"]
Oct 30 02:56:18.138042 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:56:18.138007    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6c971f_7f6a_4ed4_80aa_43d7ccce18b4.slice/crio-eab1372c7258ab35464df2ecfac6f1c67d923b5f76371c96d27d9d570cf8b58d WatchSource:0}: Error finding container eab1372c7258ab35464df2ecfac6f1c67d923b5f76371c96d27d9d570cf8b58d: Status 404 returned error can't find the container with id eab1372c7258ab35464df2ecfac6f1c67d923b5f76371c96d27d9d570cf8b58d
Oct 30 02:56:18.865346 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:18.865305    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn" event={"ID":"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4","Type":"ContainerStarted","Data":"eab1372c7258ab35464df2ecfac6f1c67d923b5f76371c96d27d9d570cf8b58d"}
Oct 30 02:56:19.870584 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:19.870550    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn" event={"ID":"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4","Type":"ContainerStarted","Data":"a756cadbca80bca9b7403a820a7c7c6d97763bfe3144aa91d125cea4b9c24573"}
Oct 30 02:56:20.876024 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:20.875986    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn" event={"ID":"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4","Type":"ContainerStarted","Data":"a538ea3711b67ec24149ade0f63c870d1c43ec62839d944c5d95311f9a6d5d2d"}
Oct 30 02:56:20.876423 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:20.876069    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn" event={"ID":"3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4","Type":"ContainerStarted","Data":"4ee2216fa5c4391796a85f7ac3f7a3ec9ea7a78ee4160475f2b01ae93add6b58"}
Oct 30 02:56:20.955242 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:20.955181    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-599ff67b9b-k8vwn" podStartSLOduration=1.801000274 podStartE2EDuration="3.955166651s" totalImagesPullingTime="2.154166377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:20.954642047 +0000 UTC m=+80.546970852" watchObservedRunningTime="2026-10-30 02:56:20.955166651 +0000 UTC m=+80.547495456"
Oct 30 02:56:21.121050 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.121019    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c6bcb5b69-qfzqh"]
Oct 30 02:56:21.123002 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.122985    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.145935 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.145905    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6bcb5b69-qfzqh"]
Oct 30 02:56:21.191526 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191497    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-serving-cert\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.191709 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191554    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbbz\" (UniqueName: \"kubernetes.io/projected/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-kube-api-access-fdbbz\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.191709 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191572    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-oauth-config\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.191709 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191640    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-tmp\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.191709 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191691    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-trusted-ca-bundle\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.191873 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191759    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-oauth-serving-cert\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.191873 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191813    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-service-ca\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.191873 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.191838    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-config\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.292923 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.292882    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-serving-cert\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293100 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.292936    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbbz\" (UniqueName: \"kubernetes.io/projected/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-kube-api-access-fdbbz\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293100 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.292961    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-oauth-config\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293100 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.292983    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-tmp\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293100 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293004    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-trusted-ca-bundle\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293100 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293027    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-oauth-serving-cert\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293100 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293053    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-service-ca\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293100 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293080    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-config\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293739 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293712    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-tmp\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293921 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293902    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-config\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293985 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293903    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-oauth-serving-cert\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.293985 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.293936    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-service-ca\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.294054 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.294017    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-trusted-ca-bundle\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.295552 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.295514    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-oauth-config\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.295552 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.295543    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-serving-cert\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.317124 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.317097    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbbz\" (UniqueName: \"kubernetes.io/projected/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-kube-api-access-fdbbz\") pod \"console-7c6bcb5b69-qfzqh\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") " pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.433184 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.433093    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:21.737699 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.737624    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c6bcb5b69-qfzqh"]
Oct 30 02:56:21.740471 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:56:21.740428    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956bc0a2_3a2e_4a78_8a6b_bac4e8352427.slice/crio-be280c4a01be2162164147472e796de4407a7dbf3335cb8583d9b61cad95547e WatchSource:0}: Error finding container be280c4a01be2162164147472e796de4407a7dbf3335cb8583d9b61cad95547e: Status 404 returned error can't find the container with id be280c4a01be2162164147472e796de4407a7dbf3335cb8583d9b61cad95547e
Oct 30 02:56:21.881330 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.881295    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bcb5b69-qfzqh" event={"ID":"956bc0a2-3a2e-4a78-8a6b-bac4e8352427","Type":"ContainerStarted","Data":"f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b"}
Oct 30 02:56:21.881851 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.881346    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bcb5b69-qfzqh" event={"ID":"956bc0a2-3a2e-4a78-8a6b-bac4e8352427","Type":"ContainerStarted","Data":"be280c4a01be2162164147472e796de4407a7dbf3335cb8583d9b61cad95547e"}
Oct 30 02:56:21.926909 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:21.926856    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7c6bcb5b69-qfzqh" podStartSLOduration=0.926841051 podStartE2EDuration="926.841051ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:21.925969933 +0000 UTC m=+81.518298740" watchObservedRunningTime="2026-10-30 02:56:21.926841051 +0000 UTC m=+81.519169860"
Oct 30 02:56:31.433885 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:31.433836    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:31.434401 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:31.433922    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:31.438634 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:31.438614    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:31.928922 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:31.928897    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:56:32.006094 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:32.006063    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666fdc54bd-kb7nv"]
Oct 30 02:56:40.258025 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:56:40.257990    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:56:57.030588 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.030510    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-666fdc54bd-kb7nv" podUID="bede845b-baec-4a7c-9fd3-eb6025701107" containerName="console" containerID="cri-o://28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4" gracePeriod=15
Oct 30 02:56:57.267052 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.267026    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:56:57.383817 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.383726    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-service-ca\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-service-ca\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.383817 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.383771    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-oauth-serving-cert\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.383817 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.383800    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-console-config\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-console-config\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.384099 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.383824    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-oauth-config\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.384099 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.383880    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bede845b-baec-4a7c-9fd3-eb6025701107-tmp\" (UniqueName: \"kubernetes.io/empty-dir/bede845b-baec-4a7c-9fd3-eb6025701107-tmp\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.384099 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.383903    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-serving-cert\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.384099 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.383944    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bede845b-baec-4a7c-9fd3-eb6025701107-kube-api-access-knwlz\" (UniqueName: \"kubernetes.io/projected/bede845b-baec-4a7c-9fd3-eb6025701107-kube-api-access-knwlz\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.384298 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384115    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-trusted-ca-bundle\") pod \"bede845b-baec-4a7c-9fd3-eb6025701107\" (UID: \"bede845b-baec-4a7c-9fd3-eb6025701107\") "
Oct 30 02:56:57.384298 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384267    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bede845b-baec-4a7c-9fd3-eb6025701107-tmp" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:56:57.384393 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384295    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-service-ca" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:57.384393 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384348    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-oauth-serving-cert" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:57.384472 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384451    4377 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bede845b-baec-4a7c-9fd3-eb6025701107-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.384589 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384572    4377 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.384657 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384597    4377 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.384710 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384683    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-trusted-ca-bundle" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:57.384710 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.384690    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-console-config" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:56:57.386133 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.386113    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-oauth-config" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:57.386224 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.386146    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bede845b-baec-4a7c-9fd3-eb6025701107-kube-api-access-knwlz" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "kube-api-access-knwlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:56:57.386224 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.386146    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-serving-cert" pod "bede845b-baec-4a7c-9fd3-eb6025701107" (UID: "bede845b-baec-4a7c-9fd3-eb6025701107"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:56:57.485569 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.485509    4377 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.485569 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.485562    4377 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.485569 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.485573    4377 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bede845b-baec-4a7c-9fd3-eb6025701107-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.485569 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.485582    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knwlz\" (UniqueName: \"kubernetes.io/projected/bede845b-baec-4a7c-9fd3-eb6025701107-kube-api-access-knwlz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:57.485878 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:57.485591    4377 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bede845b-baec-4a7c-9fd3-eb6025701107-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:56:58.007787 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.007749    4377 generic.go:356] "Generic (PLEG): container finished" podID="bede845b-baec-4a7c-9fd3-eb6025701107" containerID="28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4" exitCode=0
Oct 30 02:56:58.007987 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.007801    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-666fdc54bd-kb7nv"
Oct 30 02:56:58.007987 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.007843    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fdc54bd-kb7nv" event={"ID":"bede845b-baec-4a7c-9fd3-eb6025701107","Type":"ContainerDied","Data":"28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4"}
Oct 30 02:56:58.007987 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.007910    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-666fdc54bd-kb7nv" event={"ID":"bede845b-baec-4a7c-9fd3-eb6025701107","Type":"ContainerDied","Data":"1760a89ed47c93f8f99cc047a1849fdc0a06d9a6dd2341006a6ebb26cac6c01f"}
Oct 30 02:56:58.007987 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.007938    4377 scope.go:122] "RemoveContainer" containerID="28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4"
Oct 30 02:56:58.016295 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.016280    4377 scope.go:122] "RemoveContainer" containerID="28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4"
Oct 30 02:56:58.016591 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:56:58.016567    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4\": container with ID starting with 28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4 not found: ID does not exist" containerID="28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4"
Oct 30 02:56:58.016647 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.016602    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4"} err="failed to get container status \"28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4\": rpc error: code = NotFound desc = could not find container \"28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4\": container with ID starting with 28d1481530c0d636b88fb11fd5e28e502b0e566751b17bcd3727df50823b31f4 not found: ID does not exist"
Oct 30 02:56:58.062173 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.062141    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-666fdc54bd-kb7nv"]
Oct 30 02:56:58.075197 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.075143    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-666fdc54bd-kb7nv"]
Oct 30 02:56:58.513396 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:56:58.513361    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bede845b-baec-4a7c-9fd3-eb6025701107" path="/var/lib/kubelet/pods/bede845b-baec-4a7c-9fd3-eb6025701107/volumes"
Oct 30 02:57:10.257787 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:57:10.257751    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:57:33.779138 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.779061    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-795798578b-gxdww"]
Oct 30 02:57:33.779598 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.779382    4377 cpu_manager.go:414] "containerMap: removing container" podUID="bede845b-baec-4a7c-9fd3-eb6025701107" containerName="console"
Oct 30 02:57:33.779598 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.779396    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bede845b-baec-4a7c-9fd3-eb6025701107" containerName="console"
Oct 30 02:57:33.779598 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.779437    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="bede845b-baec-4a7c-9fd3-eb6025701107" containerName="console"
Oct 30 02:57:33.782064 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.782040    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.804284 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.804258    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795798578b-gxdww"]
Oct 30 02:57:33.851975 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.851946    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392857bc-41ef-408e-9360-6a5145dd6e5a-console-serving-cert\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.852126 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.851979    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-console-config\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.852126 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.851997    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-oauth-serving-cert\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.852126 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.852012    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-service-ca\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.852266 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.852168    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392857bc-41ef-408e-9360-6a5145dd6e5a-console-oauth-config\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.852266 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.852193    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392857bc-41ef-408e-9360-6a5145dd6e5a-tmp\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.852266 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.852226    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flhrs\" (UniqueName: \"kubernetes.io/projected/392857bc-41ef-408e-9360-6a5145dd6e5a-kube-api-access-flhrs\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.852383 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.852290    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-trusted-ca-bundle\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.953492 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953459    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392857bc-41ef-408e-9360-6a5145dd6e5a-console-oauth-config\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.953492 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953496    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392857bc-41ef-408e-9360-6a5145dd6e5a-tmp\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.953808 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953545    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flhrs\" (UniqueName: \"kubernetes.io/projected/392857bc-41ef-408e-9360-6a5145dd6e5a-kube-api-access-flhrs\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.953808 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953668    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-trusted-ca-bundle\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.953808 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953759    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392857bc-41ef-408e-9360-6a5145dd6e5a-console-serving-cert\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.953808 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953798    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-console-config\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.954002 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953826    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-oauth-serving-cert\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.954002 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953854    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-service-ca\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.954002 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.953943    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/392857bc-41ef-408e-9360-6a5145dd6e5a-tmp\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.954641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.954621    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-service-ca\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.954641 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.954630    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-console-config\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.955084 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.954671    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-oauth-serving-cert\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.955241 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.955105    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/392857bc-41ef-408e-9360-6a5145dd6e5a-trusted-ca-bundle\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.960319 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.960295    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/392857bc-41ef-408e-9360-6a5145dd6e5a-console-oauth-config\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.960404 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.960326    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/392857bc-41ef-408e-9360-6a5145dd6e5a-console-serving-cert\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:33.979667 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:33.979640    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flhrs\" (UniqueName: \"kubernetes.io/projected/392857bc-41ef-408e-9360-6a5145dd6e5a-kube-api-access-flhrs\") pod \"console-795798578b-gxdww\" (UID: \"392857bc-41ef-408e-9360-6a5145dd6e5a\") " pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:34.092495 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:34.092412    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:34.407533 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:34.407495    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-795798578b-gxdww"]
Oct 30 02:57:34.410005 ip-10-0-138-200 kubenswrapper[4377]: W1030 02:57:34.409974    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392857bc_41ef_408e_9360_6a5145dd6e5a.slice/crio-b421b915b0ebb566d7bab5d20ac4483ae859c98ab521e49308302917cbd7e8aa WatchSource:0}: Error finding container b421b915b0ebb566d7bab5d20ac4483ae859c98ab521e49308302917cbd7e8aa: Status 404 returned error can't find the container with id b421b915b0ebb566d7bab5d20ac4483ae859c98ab521e49308302917cbd7e8aa
Oct 30 02:57:35.129339 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:35.129301    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795798578b-gxdww" event={"ID":"392857bc-41ef-408e-9360-6a5145dd6e5a","Type":"ContainerStarted","Data":"57f026dbf8f38018f5be8961f52c195438fb5d0b54f5ccc1112d6496544dc3a8"}
Oct 30 02:57:35.129749 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:35.129354    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-795798578b-gxdww" event={"ID":"392857bc-41ef-408e-9360-6a5145dd6e5a","Type":"ContainerStarted","Data":"b421b915b0ebb566d7bab5d20ac4483ae859c98ab521e49308302917cbd7e8aa"}
Oct 30 02:57:35.175913 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:35.175862    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-795798578b-gxdww" podStartSLOduration=2.175849595 podStartE2EDuration="2.175849595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:35.174604645 +0000 UTC m=+154.766933450" watchObservedRunningTime="2026-10-30 02:57:35.175849595 +0000 UTC m=+154.768178460"
Oct 30 02:57:40.257844 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:57:40.257807    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:57:44.093003 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:44.092962    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:44.093421 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:44.093025    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:44.097637 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:44.097615    4377 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:44.163187 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:44.163161    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-795798578b-gxdww"
Oct 30 02:57:44.239877 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:57:44.239849    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6bcb5b69-qfzqh"]
Oct 30 02:58:09.259367 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.259306    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7c6bcb5b69-qfzqh" podUID="956bc0a2-3a2e-4a78-8a6b-bac4e8352427" containerName="console" containerID="cri-o://f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b" gracePeriod=15
Oct 30 02:58:09.500307 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.500280    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:58:09.643028 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.642986    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-service-ca\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-service-ca\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643028 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643029    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-config\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-config\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643239 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643058    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-trusted-ca-bundle\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643239 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643100    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-kube-api-access-fdbbz\" (UniqueName: \"kubernetes.io/projected/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-kube-api-access-fdbbz\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643239 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643154    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-serving-cert\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643239 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643171    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-tmp\" (UniqueName: \"kubernetes.io/empty-dir/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-tmp\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643239 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643207    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-oauth-serving-cert\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643488 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643243    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-oauth-config\") pod \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\" (UID: \"956bc0a2-3a2e-4a78-8a6b-bac4e8352427\") "
Oct 30 02:58:09.643488 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643439    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-service-ca" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:09.643624 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643490    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-trusted-ca-bundle" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:09.643787 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643755    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-oauth-serving-cert" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:09.643913 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643848    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-tmp" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 02:58:09.643913 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.643855    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-config" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 02:58:09.645467 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.645446    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-oauth-config" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:09.645576 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.645472    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-kube-api-access-fdbbz" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "kube-api-access-fdbbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 02:58:09.645576 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.645472    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-serving-cert" pod "956bc0a2-3a2e-4a78-8a6b-bac4e8352427" (UID: "956bc0a2-3a2e-4a78-8a6b-bac4e8352427"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 02:58:09.744638 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744607    4377 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-oauth-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.744638 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744634    4377 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-service-ca\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.744638 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744645    4377 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.744849 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744653    4377 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-trusted-ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.744849 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744662    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdbbz\" (UniqueName: \"kubernetes.io/projected/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-kube-api-access-fdbbz\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.744849 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744671    4377 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-console-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.744849 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744681    4377 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-tmp\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:09.744849 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:09.744689    4377 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/956bc0a2-3a2e-4a78-8a6b-bac4e8352427-oauth-serving-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 02:58:10.250058 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.250026    4377 generic.go:356] "Generic (PLEG): container finished" podID="956bc0a2-3a2e-4a78-8a6b-bac4e8352427" containerID="f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b" exitCode=0
Oct 30 02:58:10.250225 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.250077    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c6bcb5b69-qfzqh"
Oct 30 02:58:10.250225 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.250099    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bcb5b69-qfzqh" event={"ID":"956bc0a2-3a2e-4a78-8a6b-bac4e8352427","Type":"ContainerDied","Data":"f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b"}
Oct 30 02:58:10.250225 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.250139    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c6bcb5b69-qfzqh" event={"ID":"956bc0a2-3a2e-4a78-8a6b-bac4e8352427","Type":"ContainerDied","Data":"be280c4a01be2162164147472e796de4407a7dbf3335cb8583d9b61cad95547e"}
Oct 30 02:58:10.250225 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.250158    4377 scope.go:122] "RemoveContainer" containerID="f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b"
Oct 30 02:58:10.259628 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.259609    4377 scope.go:122] "RemoveContainer" containerID="f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b"
Oct 30 02:58:10.259948 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:58:10.259932    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b\": container with ID starting with f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b not found: ID does not exist" containerID="f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b"
Oct 30 02:58:10.259991 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.259961    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b"} err="failed to get container status \"f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b\": rpc error: code = NotFound desc = could not find container \"f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b\": container with ID starting with f7a793b3d1c6117ee4b3894bae1566e32d39cd6af46dd4d1f975293d4102888b not found: ID does not exist"
Oct 30 02:58:10.262984 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:58:10.262958    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:58:10.303144 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.303117    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c6bcb5b69-qfzqh"]
Oct 30 02:58:10.315999 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.315968    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c6bcb5b69-qfzqh"]
Oct 30 02:58:10.513942 ip-10-0-138-200 kubenswrapper[4377]: I1030 02:58:10.513865    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956bc0a2-3a2e-4a78-8a6b-bac4e8352427" path="/var/lib/kubelet/pods/956bc0a2-3a2e-4a78-8a6b-bac4e8352427/volumes"
Oct 30 02:58:40.258072 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:58:40.258029    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:59:10.258629 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:59:10.258589    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 02:59:40.257987 ip-10-0-138-200 kubenswrapper[4377]: E1030 02:59:40.257944    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:00:10.258122 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:00:10.258083    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:00:40.258250 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:00:40.258147    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:01:10.258067 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:10.258027    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:01:29.050719 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.050682    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"]
Oct 30 03:01:29.051342 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.051073    4377 cpu_manager.go:414] "containerMap: removing container" podUID="956bc0a2-3a2e-4a78-8a6b-bac4e8352427" containerName="console"
Oct 30 03:01:29.051342 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.051091    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956bc0a2-3a2e-4a78-8a6b-bac4e8352427" containerName="console"
Oct 30 03:01:29.051342 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.051183    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="956bc0a2-3a2e-4a78-8a6b-bac4e8352427" containerName="console"
Oct 30 03:01:29.054058 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.054042    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.081849 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.081824    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 03:01:29.081968 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.081890    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 03:01:29.081968 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.081892    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"98a21a31f7a8ae0212b9c22cc643f9202bfa37-6b211c7f-dockercfg-mfdkf\""
Oct 30 03:01:29.091595 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.091569    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"]
Oct 30 03:01:29.108727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.108702    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cgb5\" (UniqueName: \"kubernetes.io/projected/9d94aa9f-517d-47d8-88c3-111a419994ca-kube-api-access-5cgb5\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.108868 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.108747    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.108868 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.108778    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.209187 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.209157    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.209359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.209198    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.209359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.209236    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgb5\" (UniqueName: \"kubernetes.io/projected/9d94aa9f-517d-47d8-88c3-111a419994ca-kube-api-access-5cgb5\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.209580 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.209560    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.209661 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.209630    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.232885 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.232859    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cgb5\" (UniqueName: \"kubernetes.io/projected/9d94aa9f-517d-47d8-88c3-111a419994ca-kube-api-access-5cgb5\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.370093 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.370015    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:29.667726 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.667652    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"]
Oct 30 03:01:29.670878 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:01:29.670854    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d94aa9f_517d_47d8_88c3_111a419994ca.slice/crio-440addf196c6239503fa85e08e3c36d18e4bab7af8c6738f2d93e22d1d2be77b WatchSource:0}: Error finding container 440addf196c6239503fa85e08e3c36d18e4bab7af8c6738f2d93e22d1d2be77b: Status 404 returned error can't find the container with id 440addf196c6239503fa85e08e3c36d18e4bab7af8c6738f2d93e22d1d2be77b
Oct 30 03:01:29.673014 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.672999    4377 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:01:29.894048 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:29.894015    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46" event={"ID":"9d94aa9f-517d-47d8-88c3-111a419994ca","Type":"ContainerStarted","Data":"440addf196c6239503fa85e08e3c36d18e4bab7af8c6738f2d93e22d1d2be77b"}
Oct 30 03:01:35.918316 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:35.918276    4377 generic.go:356] "Generic (PLEG): container finished" podID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerID="71afd4d0f3a40a64cb5ad3ec685b63bd2fa766803a7ebfc0866e310b26a92e24" exitCode=0
Oct 30 03:01:35.918750 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:35.918363    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46" event={"ID":"9d94aa9f-517d-47d8-88c3-111a419994ca","Type":"ContainerDied","Data":"71afd4d0f3a40a64cb5ad3ec685b63bd2fa766803a7ebfc0866e310b26a92e24"}
Oct 30 03:01:38.930212 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:38.930175    4377 generic.go:356] "Generic (PLEG): container finished" podID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerID="45a158fcef8b7b6cc14ee80082a18c36957ea94606633c66e079ff8e624945a3" exitCode=0
Oct 30 03:01:38.930212 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:38.930219    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46" event={"ID":"9d94aa9f-517d-47d8-88c3-111a419994ca","Type":"ContainerDied","Data":"45a158fcef8b7b6cc14ee80082a18c36957ea94606633c66e079ff8e624945a3"}
Oct 30 03:01:40.260432 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:40.260395    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:01:44.955701 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:44.955668    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46" event={"ID":"9d94aa9f-517d-47d8-88c3-111a419994ca","Type":"ContainerStarted","Data":"edfedbb81cf54e70b11017d0f399af7c4695b74c6ccf6d82c53a35bce3ea29e9"}
Oct 30 03:01:45.000049 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:44.999991    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46" podStartSLOduration=2.956520921 podStartE2EDuration="15.999976282s" totalImagesPullingTime="13.043455361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:44.99955218 +0000 UTC m=+404.591880997" watchObservedRunningTime="2026-10-30 03:01:44.999976282 +0000 UTC m=+404.592305087"
Oct 30 03:01:45.961405 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:45.961366    4377 generic.go:356] "Generic (PLEG): container finished" podID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerID="edfedbb81cf54e70b11017d0f399af7c4695b74c6ccf6d82c53a35bce3ea29e9" exitCode=0
Oct 30 03:01:45.961880 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:45.961419    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46" event={"ID":"9d94aa9f-517d-47d8-88c3-111a419994ca","Type":"ContainerDied","Data":"edfedbb81cf54e70b11017d0f399af7c4695b74c6ccf6d82c53a35bce3ea29e9"}
Oct 30 03:01:47.087069 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.087043    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:47.159159 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.159132    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-bundle\") pod \"9d94aa9f-517d-47d8-88c3-111a419994ca\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") "
Oct 30 03:01:47.159322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.159179    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-util\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-util\") pod \"9d94aa9f-517d-47d8-88c3-111a419994ca\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") "
Oct 30 03:01:47.159322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.159222    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d94aa9f-517d-47d8-88c3-111a419994ca-kube-api-access-5cgb5\" (UniqueName: \"kubernetes.io/projected/9d94aa9f-517d-47d8-88c3-111a419994ca-kube-api-access-5cgb5\") pod \"9d94aa9f-517d-47d8-88c3-111a419994ca\" (UID: \"9d94aa9f-517d-47d8-88c3-111a419994ca\") "
Oct 30 03:01:47.159803 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.159771    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-bundle" pod "9d94aa9f-517d-47d8-88c3-111a419994ca" (UID: "9d94aa9f-517d-47d8-88c3-111a419994ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:01:47.161391 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.161359    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d94aa9f-517d-47d8-88c3-111a419994ca-kube-api-access-5cgb5" pod "9d94aa9f-517d-47d8-88c3-111a419994ca" (UID: "9d94aa9f-517d-47d8-88c3-111a419994ca"). InnerVolumeSpecName "kube-api-access-5cgb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:01:47.167679 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.167655    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-util" pod "9d94aa9f-517d-47d8-88c3-111a419994ca" (UID: "9d94aa9f-517d-47d8-88c3-111a419994ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:01:47.259815 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.259728    4377 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-util\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:01:47.259815 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.259789    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cgb5\" (UniqueName: \"kubernetes.io/projected/9d94aa9f-517d-47d8-88c3-111a419994ca-kube-api-access-5cgb5\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:01:47.259815 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.259801    4377 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d94aa9f-517d-47d8-88c3-111a419994ca-bundle\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:01:47.970278 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.970235    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46" event={"ID":"9d94aa9f-517d-47d8-88c3-111a419994ca","Type":"ContainerDied","Data":"440addf196c6239503fa85e08e3c36d18e4bab7af8c6738f2d93e22d1d2be77b"}
Oct 30 03:01:47.970278 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.970275    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820cbqt46"
Oct 30 03:01:47.970278 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:47.970287    4377 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="440addf196c6239503fa85e08e3c36d18e4bab7af8c6738f2d93e22d1d2be77b"
Oct 30 03:01:51.565407 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565371    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"]
Oct 30 03:01:51.565887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565673    4377 cpu_manager.go:414] "containerMap: removing container" podUID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerName="pull"
Oct 30 03:01:51.565887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565685    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerName="pull"
Oct 30 03:01:51.565887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565694    4377 cpu_manager.go:414] "containerMap: removing container" podUID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerName="extract"
Oct 30 03:01:51.565887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565699    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerName="extract"
Oct 30 03:01:51.565887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565716    4377 cpu_manager.go:414] "containerMap: removing container" podUID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerName="util"
Oct 30 03:01:51.565887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565723    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerName="util"
Oct 30 03:01:51.565887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.565765    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d94aa9f-517d-47d8-88c3-111a419994ca" containerName="extract"
Oct 30 03:01:51.605047 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.605013    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"]
Oct 30 03:01:51.605240 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.605139    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:51.621582 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.621551    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Oct 30 03:01:51.621754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.621652    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Oct 30 03:01:51.621754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.621690    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Oct 30 03:01:51.636129 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.636100    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-p79fw\""
Oct 30 03:01:51.690803 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.690768    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40de4f87-d07e-44d2-a3b1-66df2932c630-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-4kgrc\" (UID: \"40de4f87-d07e-44d2-a3b1-66df2932c630\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:51.690982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.690824    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flslp\" (UniqueName: \"kubernetes.io/projected/40de4f87-d07e-44d2-a3b1-66df2932c630-kube-api-access-flslp\") pod \"custom-metrics-autoscaler-operator-c667576c8-4kgrc\" (UID: \"40de4f87-d07e-44d2-a3b1-66df2932c630\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:51.792208 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.792167    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40de4f87-d07e-44d2-a3b1-66df2932c630-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-4kgrc\" (UID: \"40de4f87-d07e-44d2-a3b1-66df2932c630\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:51.792426 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.792229    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flslp\" (UniqueName: \"kubernetes.io/projected/40de4f87-d07e-44d2-a3b1-66df2932c630-kube-api-access-flslp\") pod \"custom-metrics-autoscaler-operator-c667576c8-4kgrc\" (UID: \"40de4f87-d07e-44d2-a3b1-66df2932c630\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:51.794619 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.794597    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/40de4f87-d07e-44d2-a3b1-66df2932c630-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-4kgrc\" (UID: \"40de4f87-d07e-44d2-a3b1-66df2932c630\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:51.818157 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.818095    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flslp\" (UniqueName: \"kubernetes.io/projected/40de4f87-d07e-44d2-a3b1-66df2932c630-kube-api-access-flslp\") pod \"custom-metrics-autoscaler-operator-c667576c8-4kgrc\" (UID: \"40de4f87-d07e-44d2-a3b1-66df2932c630\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:51.929994 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:51.929954    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:52.281500 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:52.281469    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"]
Oct 30 03:01:52.297928 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:01:52.297892    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40de4f87_d07e_44d2_a3b1_66df2932c630.slice/crio-0d5774eb625a35472fad3e96ca5995b312ed24cf2691528fcd2e74c6732d9b72 WatchSource:0}: Error finding container 0d5774eb625a35472fad3e96ca5995b312ed24cf2691528fcd2e74c6732d9b72: Status 404 returned error can't find the container with id 0d5774eb625a35472fad3e96ca5995b312ed24cf2691528fcd2e74c6732d9b72
Oct 30 03:01:52.988843 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:52.988808    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc" event={"ID":"40de4f87-d07e-44d2-a3b1-66df2932c630","Type":"ContainerStarted","Data":"0d5774eb625a35472fad3e96ca5995b312ed24cf2691528fcd2e74c6732d9b72"}
Oct 30 03:01:57.006894 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.006860    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc" event={"ID":"40de4f87-d07e-44d2-a3b1-66df2932c630","Type":"ContainerStarted","Data":"f8225449ed2989e67c38c27e2b5dfa61e3332db6ab102dcbc749e28d9e2d3399"}
Oct 30 03:01:57.007444 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.006982    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:01:57.107103 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.107052    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc" podStartSLOduration=2.297093126 podStartE2EDuration="6.107035943s" totalImagesPullingTime="3.809942817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:01:57.09350224 +0000 UTC m=+416.685831046" watchObservedRunningTime="2026-10-30 03:01:57.107035943 +0000 UTC m=+416.699364749"
Oct 30 03:01:57.397120 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.397085    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-7dc85965f-x9p6m"]
Oct 30 03:01:57.418691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.418642    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.421573 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.421549    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-x9p6m"]
Oct 30 03:01:57.434217 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.434185    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Oct 30 03:01:57.507866 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.507829    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Oct 30 03:01:57.508050 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.507883    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-c57l9\""
Oct 30 03:01:57.544565 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.544533    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c56d1e23-22f1-4135-b72c-a8b296c58fe1-cabundle0\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.544727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.544581    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4g4n\" (UniqueName: \"kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-kube-api-access-c4g4n\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.544727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.544692    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-certificates\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.645166 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.645126    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c56d1e23-22f1-4135-b72c-a8b296c58fe1-cabundle0\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.645370 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.645202    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4g4n\" (UniqueName: \"kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-kube-api-access-c4g4n\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.645370 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.645252    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-certificates\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.645729 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:57.645676    4377 secret.go:281] references non-existent secret key: ca.crt
Oct 30 03:01:57.645729 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:57.645703    4377 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Oct 30 03:01:57.645729 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:57.645715    4377 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-x9p6m: references non-existent secret key: ca.crt
Oct 30 03:01:57.646010 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:57.645791    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-certificates podName:c56d1e23-22f1-4135-b72c-a8b296c58fe1 nodeName:}" failed. No retries permitted until 2026-10-30 03:01:58.145770042 +0000 UTC m=+417.738098839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-certificates") pod "keda-operator-7dc85965f-x9p6m" (UID: "c56d1e23-22f1-4135-b72c-a8b296c58fe1") : references non-existent secret key: ca.crt
Oct 30 03:01:57.646010 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.645964    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/c56d1e23-22f1-4135-b72c-a8b296c58fe1-cabundle0\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:57.670998 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:57.670936    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4g4n\" (UniqueName: \"kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-kube-api-access-c4g4n\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:58.149691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.149651    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-certificates\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:58.152119 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.152093    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c56d1e23-22f1-4135-b72c-a8b296c58fe1-certificates\") pod \"keda-operator-7dc85965f-x9p6m\" (UID: \"c56d1e23-22f1-4135-b72c-a8b296c58fe1\") " pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:58.174918 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.174891    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"]
Oct 30 03:01:58.177172 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.177150    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.193971 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.193945    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Oct 30 03:01:58.199687 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.199664    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"]
Oct 30 03:01:58.251002 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.250976    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/493dd738-45ac-441b-ac5f-c00381227982-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.251122 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.251013    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/493dd738-45ac-441b-ac5f-c00381227982-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.251167 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.251153    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25dcq\" (UniqueName: \"kubernetes.io/projected/493dd738-45ac-441b-ac5f-c00381227982-kube-api-access-25dcq\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.342652 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.342623    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:01:58.351639 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.351607    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25dcq\" (UniqueName: \"kubernetes.io/projected/493dd738-45ac-441b-ac5f-c00381227982-kube-api-access-25dcq\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.351778 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.351660    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/493dd738-45ac-441b-ac5f-c00381227982-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.351855 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.351813    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/493dd738-45ac-441b-ac5f-c00381227982-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.352352 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.352323    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/493dd738-45ac-441b-ac5f-c00381227982-temp-vol\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.354549 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.354506    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/493dd738-45ac-441b-ac5f-c00381227982-certificates\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.375330 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.375304    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25dcq\" (UniqueName: \"kubernetes.io/projected/493dd738-45ac-441b-ac5f-c00381227982-kube-api-access-25dcq\") pod \"keda-metrics-apiserver-86bd856cfb-cmqqf\" (UID: \"493dd738-45ac-441b-ac5f-c00381227982\") " pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.461392 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.461361    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-z2jgz"]
Oct 30 03:01:58.464358 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.464339    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:58.484943 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.484917    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Oct 30 03:01:58.486622 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.486595    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-z2jgz"]
Oct 30 03:01:58.488098 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.488075    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:01:58.554104 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.554068    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-certificates\") pod \"keda-admission-7b7c985c58-z2jgz\" (UID: \"fc3008b6-e148-41d2-a440-be575924a632\") " pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:58.554269 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.554127    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6bl8\" (UniqueName: \"kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-kube-api-access-g6bl8\") pod \"keda-admission-7b7c985c58-z2jgz\" (UID: \"fc3008b6-e148-41d2-a440-be575924a632\") " pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:58.647882 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.647743    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-x9p6m"]
Oct 30 03:01:58.650472 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:01:58.650410    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56d1e23_22f1_4135_b72c_a8b296c58fe1.slice/crio-5d47886c5f3a51355ee14c2db8ad13f3ba8dce976f8902fa312e4676df92d6fd WatchSource:0}: Error finding container 5d47886c5f3a51355ee14c2db8ad13f3ba8dce976f8902fa312e4676df92d6fd: Status 404 returned error can't find the container with id 5d47886c5f3a51355ee14c2db8ad13f3ba8dce976f8902fa312e4676df92d6fd
Oct 30 03:01:58.654772 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.654743    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-certificates\") pod \"keda-admission-7b7c985c58-z2jgz\" (UID: \"fc3008b6-e148-41d2-a440-be575924a632\") " pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:58.654868 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.654791    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6bl8\" (UniqueName: \"kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-kube-api-access-g6bl8\") pod \"keda-admission-7b7c985c58-z2jgz\" (UID: \"fc3008b6-e148-41d2-a440-be575924a632\") " pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:58.654912 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:58.654873    4377 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Oct 30 03:01:58.654912 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:58.654901    4377 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-7b7c985c58-z2jgz: secret "keda-admission-webhooks-certs" not found
Oct 30 03:01:58.654977 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:01:58.654944    4377 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-certificates podName:fc3008b6-e148-41d2-a440-be575924a632 nodeName:}" failed. No retries permitted until 2026-10-30 03:01:59.154930102 +0000 UTC m=+418.747258886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-certificates") pod "keda-admission-7b7c985c58-z2jgz" (UID: "fc3008b6-e148-41d2-a440-be575924a632") : secret "keda-admission-webhooks-certs" not found
Oct 30 03:01:58.701446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.701419    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6bl8\" (UniqueName: \"kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-kube-api-access-g6bl8\") pod \"keda-admission-7b7c985c58-z2jgz\" (UID: \"fc3008b6-e148-41d2-a440-be575924a632\") " pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:58.787046 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:58.786870    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"]
Oct 30 03:01:58.789572 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:01:58.789541    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493dd738_45ac_441b_ac5f_c00381227982.slice/crio-1733aca18bc5d3f4912629ab238ad4f87cd9b7e99f06a226ec56ef5a986a92ca WatchSource:0}: Error finding container 1733aca18bc5d3f4912629ab238ad4f87cd9b7e99f06a226ec56ef5a986a92ca: Status 404 returned error can't find the container with id 1733aca18bc5d3f4912629ab238ad4f87cd9b7e99f06a226ec56ef5a986a92ca
Oct 30 03:01:59.016275 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:59.016178    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf" event={"ID":"493dd738-45ac-441b-ac5f-c00381227982","Type":"ContainerStarted","Data":"1733aca18bc5d3f4912629ab238ad4f87cd9b7e99f06a226ec56ef5a986a92ca"}
Oct 30 03:01:59.017304 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:59.017278    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-x9p6m" event={"ID":"c56d1e23-22f1-4135-b72c-a8b296c58fe1","Type":"ContainerStarted","Data":"5d47886c5f3a51355ee14c2db8ad13f3ba8dce976f8902fa312e4676df92d6fd"}
Oct 30 03:01:59.158593 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:59.158559    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-certificates\") pod \"keda-admission-7b7c985c58-z2jgz\" (UID: \"fc3008b6-e148-41d2-a440-be575924a632\") " pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:59.160919 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:59.160892    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/fc3008b6-e148-41d2-a440-be575924a632-certificates\") pod \"keda-admission-7b7c985c58-z2jgz\" (UID: \"fc3008b6-e148-41d2-a440-be575924a632\") " pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:59.375482 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:59.375446    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:01:59.672199 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:01:59.672173    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-7b7c985c58-z2jgz"]
Oct 30 03:01:59.674960 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:01:59.674936    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3008b6_e148_41d2_a440_be575924a632.slice/crio-e6259cb89d7afcee9f181f88ef701404569ef534a7a0355eb60f8f8972d2254a WatchSource:0}: Error finding container e6259cb89d7afcee9f181f88ef701404569ef534a7a0355eb60f8f8972d2254a: Status 404 returned error can't find the container with id e6259cb89d7afcee9f181f88ef701404569ef534a7a0355eb60f8f8972d2254a
Oct 30 03:02:00.023639 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:00.023559    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-z2jgz" event={"ID":"fc3008b6-e148-41d2-a440-be575924a632","Type":"ContainerStarted","Data":"e6259cb89d7afcee9f181f88ef701404569ef534a7a0355eb60f8f8972d2254a"}
Oct 30 03:02:02.034583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:02.034544    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-7b7c985c58-z2jgz" event={"ID":"fc3008b6-e148-41d2-a440-be575924a632","Type":"ContainerStarted","Data":"af03a748532438c76f0c43cfa156398e08d9a33c9ae80b1184f4a5518247a158"}
Oct 30 03:02:02.035074 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:02.034649    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:02:02.077291 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:02.077243    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-admission-7b7c985c58-z2jgz" podStartSLOduration=2.42960436 podStartE2EDuration="4.077228718s" totalImagesPullingTime="1.647624358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:02.076113012 +0000 UTC m=+421.668441820" watchObservedRunningTime="2026-10-30 03:02:02.077228718 +0000 UTC m=+421.669557514"
Oct 30 03:02:03.040365 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:03.040330    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf" event={"ID":"493dd738-45ac-441b-ac5f-c00381227982","Type":"ContainerStarted","Data":"a137de7475d744df221a4c1a9fe266ab85bc008b86e883c5edb323ab4abc37a1"}
Oct 30 03:02:03.040800 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:03.040543    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:02:03.084077 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:03.084032    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf" podStartSLOduration=1.7848266609999999 podStartE2EDuration="5.084016708s" totalImagesPullingTime="3.299190047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:03.082511937 +0000 UTC m=+422.674840755" watchObservedRunningTime="2026-10-30 03:02:03.084016708 +0000 UTC m=+422.676345513"
Oct 30 03:02:10.264641 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:02:10.264512    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:02:12.077097 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:12.077059    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-x9p6m" event={"ID":"c56d1e23-22f1-4135-b72c-a8b296c58fe1","Type":"ContainerStarted","Data":"c06b8be982bdb4cf3f2b4a35dda6af248cdca8af428847c3c4a3e5d58f9b044b"}
Oct 30 03:02:12.077544 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:12.077169    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:02:12.119614 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:12.119565    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-operator-7dc85965f-x9p6m" podStartSLOduration=2.567053119 podStartE2EDuration="15.119551367s" totalImagesPullingTime="12.552498248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:02:12.118642598 +0000 UTC m=+431.710971405" watchObservedRunningTime="2026-10-30 03:02:12.119551367 +0000 UTC m=+431.711880170"
Oct 30 03:02:14.048357 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:14.048317    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-86bd856cfb-cmqqf"
Oct 30 03:02:18.013700 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:18.013671    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-4kgrc"
Oct 30 03:02:23.043177 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:23.043147    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-7b7c985c58-z2jgz"
Oct 30 03:02:33.083650 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:02:33.083617    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-7dc85965f-x9p6m"
Oct 30 03:02:40.258950 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:02:40.258917    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:03:10.257491 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:03:10.257456    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:03:26.758371 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.758294    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-59b8df5579-lfwjx"]
Oct 30 03:03:26.760293 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.760275    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:26.782867 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.782830    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-59b8df5579-lfwjx"]
Oct 30 03:03:26.836212 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.836180    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Oct 30 03:03:26.836212 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.836209    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-nxgdq\""
Oct 30 03:03:26.838939 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.838922    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Oct 30 03:03:26.839098 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.839084    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Oct 30 03:03:26.863538 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.863501    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ae8117-91fb-4203-8bc9-2247c0e316ca-cert\") pod \"llmisvc-controller-manager-59b8df5579-lfwjx\" (UID: \"b2ae8117-91fb-4203-8bc9-2247c0e316ca\") " pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:26.863652 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.863571    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xgs5\" (UniqueName: \"kubernetes.io/projected/b2ae8117-91fb-4203-8bc9-2247c0e316ca-kube-api-access-4xgs5\") pod \"llmisvc-controller-manager-59b8df5579-lfwjx\" (UID: \"b2ae8117-91fb-4203-8bc9-2247c0e316ca\") " pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:26.964124 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.964097    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xgs5\" (UniqueName: \"kubernetes.io/projected/b2ae8117-91fb-4203-8bc9-2247c0e316ca-kube-api-access-4xgs5\") pod \"llmisvc-controller-manager-59b8df5579-lfwjx\" (UID: \"b2ae8117-91fb-4203-8bc9-2247c0e316ca\") " pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:26.964243 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.964193    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ae8117-91fb-4203-8bc9-2247c0e316ca-cert\") pod \"llmisvc-controller-manager-59b8df5579-lfwjx\" (UID: \"b2ae8117-91fb-4203-8bc9-2247c0e316ca\") " pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:26.966646 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.966625    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b2ae8117-91fb-4203-8bc9-2247c0e316ca-cert\") pod \"llmisvc-controller-manager-59b8df5579-lfwjx\" (UID: \"b2ae8117-91fb-4203-8bc9-2247c0e316ca\") " pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:26.988394 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:26.988366    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xgs5\" (UniqueName: \"kubernetes.io/projected/b2ae8117-91fb-4203-8bc9-2247c0e316ca-kube-api-access-4xgs5\") pod \"llmisvc-controller-manager-59b8df5579-lfwjx\" (UID: \"b2ae8117-91fb-4203-8bc9-2247c0e316ca\") " pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:27.072468 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:27.072430    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:27.366121 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:27.366038    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-59b8df5579-lfwjx"]
Oct 30 03:03:27.370073 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:03:27.370048    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ae8117_91fb_4203_8bc9_2247c0e316ca.slice/crio-e24862a914e178fa973e46ba4afdfe10496855b191ecf989d418fd96c68d0cac WatchSource:0}: Error finding container e24862a914e178fa973e46ba4afdfe10496855b191ecf989d418fd96c68d0cac: Status 404 returned error can't find the container with id e24862a914e178fa973e46ba4afdfe10496855b191ecf989d418fd96c68d0cac
Oct 30 03:03:28.376299 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:28.376257    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx" event={"ID":"b2ae8117-91fb-4203-8bc9-2247c0e316ca","Type":"ContainerStarted","Data":"e24862a914e178fa973e46ba4afdfe10496855b191ecf989d418fd96c68d0cac"}
Oct 30 03:03:30.385571 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:30.385514    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx" event={"ID":"b2ae8117-91fb-4203-8bc9-2247c0e316ca","Type":"ContainerStarted","Data":"43f80e8845e3db9baa72b0471d0500e25d594490cca2a158500bb04428f55f40"}
Oct 30 03:03:30.385983 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:30.385644    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:03:30.429169 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:03:30.429122    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx" podStartSLOduration=1.952559129 podStartE2EDuration="4.429107301s" totalImagesPullingTime="2.476548172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:03:30.428006427 +0000 UTC m=+510.020335235" watchObservedRunningTime="2026-10-30 03:03:30.429107301 +0000 UTC m=+510.021436128"
Oct 30 03:03:40.259083 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:03:40.259041    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:04:01.393025 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:01.392996    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-59b8df5579-lfwjx"
Oct 30 03:04:10.258598 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:04:10.258556    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:04:40.258514 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:04:40.258411    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:04:50.662621 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.662536    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-scvd2"]
Oct 30 03:04:50.665810 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.665790    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.685836 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.685811    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-scvd2"]
Oct 30 03:04:50.704087 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.704064    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqs2\" (UniqueName: \"kubernetes.io/projected/f789d534-ec04-418c-add4-a164d62a3143-kube-api-access-2dqs2\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.704223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.704197    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f789d534-ec04-418c-add4-a164d62a3143-cert\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.704288 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.704225    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f789d534-ec04-418c-add4-a164d62a3143-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.722692 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.722678    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-f2dvg\""
Oct 30 03:04:50.722834 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.722728    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Oct 30 03:04:50.722834 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.722779    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Oct 30 03:04:50.805175 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.805128    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqs2\" (UniqueName: \"kubernetes.io/projected/f789d534-ec04-418c-add4-a164d62a3143-kube-api-access-2dqs2\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.805340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.805240    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f789d534-ec04-418c-add4-a164d62a3143-cert\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.805340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.805266    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f789d534-ec04-418c-add4-a164d62a3143-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.807709 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.807678    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f789d534-ec04-418c-add4-a164d62a3143-metrics-certs\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.807709 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.807684    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f789d534-ec04-418c-add4-a164d62a3143-cert\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.829504 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.829476    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqs2\" (UniqueName: \"kubernetes.io/projected/f789d534-ec04-418c-add4-a164d62a3143-kube-api-access-2dqs2\") pod \"odh-model-controller-7f9f8cdc65-scvd2\" (UID: \"f789d534-ec04-418c-add4-a164d62a3143\") " pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:50.978100 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:50.978015    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:51.273082 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:51.273005    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7f9f8cdc65-scvd2"]
Oct 30 03:04:51.275960 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:04:51.275934    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf789d534_ec04_418c_add4_a164d62a3143.slice/crio-0c2f4cf54eb2acedaa8fd26ce3cb4756219fcb4302895c1b4536b890bb0eb043 WatchSource:0}: Error finding container 0c2f4cf54eb2acedaa8fd26ce3cb4756219fcb4302895c1b4536b890bb0eb043: Status 404 returned error can't find the container with id 0c2f4cf54eb2acedaa8fd26ce3cb4756219fcb4302895c1b4536b890bb0eb043
Oct 30 03:04:51.690112 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:51.690066    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-scvd2" event={"ID":"f789d534-ec04-418c-add4-a164d62a3143","Type":"ContainerStarted","Data":"0c2f4cf54eb2acedaa8fd26ce3cb4756219fcb4302895c1b4536b890bb0eb043"}
Oct 30 03:04:53.700292 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:53.700255    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7f9f8cdc65-scvd2" event={"ID":"f789d534-ec04-418c-add4-a164d62a3143","Type":"ContainerStarted","Data":"15f9364405b429d28ec59d21b513c82e65582c74746fdc027f65256dc48e2162"}
Oct 30 03:04:53.700766 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:53.700402    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:04:53.742827 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:04:53.742767    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/odh-model-controller-7f9f8cdc65-scvd2" podStartSLOduration=1.413858168 podStartE2EDuration="3.742749863s" totalImagesPullingTime="2.328891695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:04:53.742359048 +0000 UTC m=+593.334687856" watchObservedRunningTime="2026-10-30 03:04:53.742749863 +0000 UTC m=+593.335078672"
Oct 30 03:05:04.706675 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:05:04.706640    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7f9f8cdc65-scvd2"
Oct 30 03:05:10.257989 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:05:10.257949    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:05:40.258581 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:05:40.258545    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:06:10.258364 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:06:10.258323    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:06:40.258589 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:06:40.258552    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:06:46.331588 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.331546    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-bf7sj"]
Oct 30 03:06:46.333876 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.333854    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.352646 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.352619    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Oct 30 03:06:46.352849 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.352828    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-55fdx\""
Oct 30 03:06:46.352926 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.352895    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Oct 30 03:06:46.354145 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.354119    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-bf7sj"]
Oct 30 03:06:46.439706 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.439675    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/773e772e-57eb-4db8-8d72-1e393b11b1f0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.439706 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.439711    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/773e772e-57eb-4db8-8d72-1e393b11b1f0-data\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.439936 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.439730    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl7td\" (UniqueName: \"kubernetes.io/projected/773e772e-57eb-4db8-8d72-1e393b11b1f0-kube-api-access-jl7td\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.540386 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.540353    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/773e772e-57eb-4db8-8d72-1e393b11b1f0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.540386 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.540383    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/773e772e-57eb-4db8-8d72-1e393b11b1f0-data\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.540619 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.540405    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl7td\" (UniqueName: \"kubernetes.io/projected/773e772e-57eb-4db8-8d72-1e393b11b1f0-kube-api-access-jl7td\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.540905 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.540885    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/773e772e-57eb-4db8-8d72-1e393b11b1f0-data\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.542816 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.542791    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/773e772e-57eb-4db8-8d72-1e393b11b1f0-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.564686 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.564663    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl7td\" (UniqueName: \"kubernetes.io/projected/773e772e-57eb-4db8-8d72-1e393b11b1f0-kube-api-access-jl7td\") pod \"seaweedfs-tls-serving-5c74974785-bf7sj\" (UID: \"773e772e-57eb-4db8-8d72-1e393b11b1f0\") " pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.644469 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.644388    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj"
Oct 30 03:06:46.943511 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.943486    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-5c74974785-bf7sj"]
Oct 30 03:06:46.946088 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:06:46.946049    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773e772e_57eb_4db8_8d72_1e393b11b1f0.slice/crio-0c1930b6aae08a9deacfe738eb05facacd155d857d3c1464beb9f3088354e690 WatchSource:0}: Error finding container 0c1930b6aae08a9deacfe738eb05facacd155d857d3c1464beb9f3088354e690: Status 404 returned error can't find the container with id 0c1930b6aae08a9deacfe738eb05facacd155d857d3c1464beb9f3088354e690
Oct 30 03:06:46.947433 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:46.947416    4377 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:06:47.135554 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:47.135501    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj" event={"ID":"773e772e-57eb-4db8-8d72-1e393b11b1f0","Type":"ContainerStarted","Data":"0c1930b6aae08a9deacfe738eb05facacd155d857d3c1464beb9f3088354e690"}
Oct 30 03:06:50.152988 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:50.152947    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj" event={"ID":"773e772e-57eb-4db8-8d72-1e393b11b1f0","Type":"ContainerStarted","Data":"3d8a74b6ee32b776d3f3dccf3db4f158e83f923ebb1cf93713595a76b34ae227"}
Oct 30 03:06:50.195047 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:06:50.194990    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-5c74974785-bf7sj" podStartSLOduration=1.5173386089999998 podStartE2EDuration="4.194973823s" totalImagesPullingTime="2.677635214s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:06:50.194378991 +0000 UTC m=+709.786707798" watchObservedRunningTime="2026-10-30 03:06:50.194973823 +0000 UTC m=+709.787302630"
Oct 30 03:07:10.258664 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:07:10.258624    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:07:40.259239 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:07:40.259200    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:08:10.258512 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:08:10.258469    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:08:40.259342 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:08:40.259301    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:09:10.258300 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:09:10.258260    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:09:40.258253 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:09:40.258216    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:10:10.259120 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:10:10.259083    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:10:40.258206 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:10:40.258165    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:11:10.257679 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:11:10.257639    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:11:40.259178 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:11:40.259140    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:12:10.258317 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:12:10.258282    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:12:40.258712 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:12:40.258619    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:13:10.258750 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:13:10.258707    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:13:40.259115 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:13:40.259074    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:14:10.258301 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:14:10.258260    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:14:40.258819 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:14:40.258776    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:15:10.259550 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:15:10.259495    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:15:40.259101 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:15:40.259019    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:16:10.258558 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:16:10.258505    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:16:40.258556 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:16:40.258505    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:17:10.258943 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:17:10.258905    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:17:40.258368 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:17:40.258325    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:18:10.258765 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:18:10.258720    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:18:40.259304 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:18:40.259223    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:19:10.257697 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:19:10.257655    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:19:40.258475 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:19:40.258438    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:20:10.259195 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:20:10.259152    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:20:40.258446 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:20:40.258405    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:21:10.257910 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:21:10.257873    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:21:40.258404 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:21:40.258314    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:22:10.259351 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:22:10.259308    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:22:40.259333 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:22:40.259296    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:23:10.258362 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:23:10.258315    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:23:40.258967 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:23:40.258926    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:24:10.259180 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:24:10.259141    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:24:40.259558 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:24:40.259508    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:25:02.362826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.362789    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"]
Oct 30 03:25:02.366413 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.366386    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.396000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.395965    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:25:02.396000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.395981    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:25:02.396224 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.396023    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Oct 30 03:25:02.396224 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.396020    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:25:02.396224 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.396043    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Oct 30 03:25:02.396224 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.396045    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:25:02.396224 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.396075    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:25:02.397586 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.397555    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"]
Oct 30 03:25:02.545881 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.545837    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.545881 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.545886    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.546101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.545999    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-proxy-tls\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.546101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.546028    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnhsg\" (UniqueName: \"kubernetes.io/projected/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-api-access-gnhsg\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.546101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.546094    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.647290 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.647202    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-proxy-tls\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.647462 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.647424    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnhsg\" (UniqueName: \"kubernetes.io/projected/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-api-access-gnhsg\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.647547 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.647484    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.647610 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.647574    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.647666 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.647617    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.647999 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.647977    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.648154 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.648137    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.648193 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.648180    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.649707 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.649683    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-proxy-tls\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.671757 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.671729    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnhsg\" (UniqueName: \"kubernetes.io/projected/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-api-access-gnhsg\") pod \"isvc-autogluon-v1-predictor-c58d788b6-k4d26\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:02.683567 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:02.683541    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:25:03.000177 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:03.000152    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"]
Oct 30 03:25:03.002470 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:25:03.002440    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe2333f_ba43_4ac6_8a9c_d2016371e28f.slice/crio-05e1577c07a4310825f2020a726c08baf8336495a7ee82d92a32e87659ec458e WatchSource:0}: Error finding container 05e1577c07a4310825f2020a726c08baf8336495a7ee82d92a32e87659ec458e: Status 404 returned error can't find the container with id 05e1577c07a4310825f2020a726c08baf8336495a7ee82d92a32e87659ec458e
Oct 30 03:25:03.004787 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:03.004769    4377 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:25:03.387053 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:03.387018    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerStarted","Data":"05e1577c07a4310825f2020a726c08baf8336495a7ee82d92a32e87659ec458e"}
Oct 30 03:25:07.407265 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:07.407227    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerStarted","Data":"9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254"}
Oct 30 03:25:10.259090 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:25:10.259047    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:25:36.522827 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:36.522792    4377 generic.go:356] "Generic (PLEG): container finished" podID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerID="9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254" exitCode=0
Oct 30 03:25:36.523251 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:25:36.522868    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerDied","Data":"9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254"}
Oct 30 03:25:40.263082 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:25:40.263030    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:26:10.262097 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:26:10.262058    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:26:40.262715 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:26:40.262675    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:27:10.262674 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:27:10.262633    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:27:29.028966 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:29.028928    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerStarted","Data":"33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9"}
Oct 30 03:27:33.047227 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:33.047191    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerStarted","Data":"2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072"}
Oct 30 03:27:33.047643 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:33.047324    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:27:33.111711 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:33.111641    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podStartSLOduration=2.887435275 podStartE2EDuration="2m31.111620303s" totalImagesPullingTime="1m59.118327441s" totalInitContainerRuntime="29.105857587s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:33.110923191 +0000 UTC m=+1952.703251998" watchObservedRunningTime="2026-10-30 03:27:33.111620303 +0000 UTC m=+1952.703949109"
Oct 30 03:27:34.051775 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:34.051746    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:27:34.053090 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:34.053055    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 03:27:35.055646 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:35.055600    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 03:27:40.061581 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:40.061549    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:27:40.062275 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:40.062242    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 03:27:40.259882 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:27:40.259838    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:27:50.063571 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:50.063542    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:27:58.943159 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:58.943124    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"]
Oct 30 03:27:58.943765 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:58.943424    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container" containerID="cri-o://33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9" gracePeriod=30
Oct 30 03:27:58.943765 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:58.943482    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kube-rbac-proxy" containerID="cri-o://2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072" gracePeriod=30
Oct 30 03:27:59.158923 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:59.158888    4377 generic.go:356] "Generic (PLEG): container finished" podID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerID="2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072" exitCode=0
Oct 30 03:27:59.159094 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:27:59.158966    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerDied","Data":"2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072"}
Oct 30 03:28:00.056322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.056275    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Oct 30 03:28:00.063152 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.063120    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Oct 30 03:28:00.322348 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.322285    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:28:00.384445 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.384410    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-cabundle-cert\") pod \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") "
Oct 30 03:28:00.384628 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.384573    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-rbac-proxy-sar-config\") pod \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") "
Oct 30 03:28:00.384628 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.384612    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-api-access-gnhsg\" (UniqueName: \"kubernetes.io/projected/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-api-access-gnhsg\") pod \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") "
Oct 30 03:28:00.384728 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.384650    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-proxy-tls\") pod \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") "
Oct 30 03:28:00.384728 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.384697    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kserve-provision-location\") pod \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\" (UID: \"1fe2333f-ba43-4ac6-8a9c-d2016371e28f\") "
Oct 30 03:28:00.384901 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.384873    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-cabundle-cert" pod "1fe2333f-ba43-4ac6-8a9c-d2016371e28f" (UID: "1fe2333f-ba43-4ac6-8a9c-d2016371e28f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:00.384975 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.384882    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-rbac-proxy-sar-config" pod "1fe2333f-ba43-4ac6-8a9c-d2016371e28f" (UID: "1fe2333f-ba43-4ac6-8a9c-d2016371e28f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:28:00.387172 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.387135    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-proxy-tls" pod "1fe2333f-ba43-4ac6-8a9c-d2016371e28f" (UID: "1fe2333f-ba43-4ac6-8a9c-d2016371e28f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:28:00.387283 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.387204    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-api-access-gnhsg" pod "1fe2333f-ba43-4ac6-8a9c-d2016371e28f" (UID: "1fe2333f-ba43-4ac6-8a9c-d2016371e28f"). InnerVolumeSpecName "kube-api-access-gnhsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:28:00.390154 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.390125    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kserve-provision-location" pod "1fe2333f-ba43-4ac6-8a9c-d2016371e28f" (UID: "1fe2333f-ba43-4ac6-8a9c-d2016371e28f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:28:00.486145 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.486110    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:28:00.486145 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.486142    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:28:00.486145 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.486151    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:28:00.486402 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.486160    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnhsg\" (UniqueName: \"kubernetes.io/projected/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-kube-api-access-gnhsg\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:28:00.486402 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:00.486169    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1fe2333f-ba43-4ac6-8a9c-d2016371e28f-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:28:01.169897 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.169860    4377 generic.go:356] "Generic (PLEG): container finished" podID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerID="33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9" exitCode=0
Oct 30 03:28:01.170325 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.169896    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerDied","Data":"33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9"}
Oct 30 03:28:01.170325 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.169948    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"
Oct 30 03:28:01.170325 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.169971    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26" event={"ID":"1fe2333f-ba43-4ac6-8a9c-d2016371e28f","Type":"ContainerDied","Data":"05e1577c07a4310825f2020a726c08baf8336495a7ee82d92a32e87659ec458e"}
Oct 30 03:28:01.170325 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.170012    4377 scope.go:122] "RemoveContainer" containerID="2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072"
Oct 30 03:28:01.177983 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.177965    4377 scope.go:122] "RemoveContainer" containerID="33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9"
Oct 30 03:28:01.185462 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.185140    4377 scope.go:122] "RemoveContainer" containerID="9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254"
Oct 30 03:28:01.194062 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.194040    4377 scope.go:122] "RemoveContainer" containerID="2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072"
Oct 30 03:28:01.194336 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:28:01.194317    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072\": container with ID starting with 2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072 not found: ID does not exist" containerID="2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072"
Oct 30 03:28:01.194411 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.194345    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072"} err="failed to get container status \"2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072\": rpc error: code = NotFound desc = could not find container \"2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072\": container with ID starting with 2285b7a4d8e79a5f4566eb93add76988ae6671ff8f7126dff0ba03fe86895072 not found: ID does not exist"
Oct 30 03:28:01.194411 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.194361    4377 scope.go:122] "RemoveContainer" containerID="33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9"
Oct 30 03:28:01.194768 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:28:01.194743    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9\": container with ID starting with 33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9 not found: ID does not exist" containerID="33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9"
Oct 30 03:28:01.194822 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.194774    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9"} err="failed to get container status \"33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9\": rpc error: code = NotFound desc = could not find container \"33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9\": container with ID starting with 33f7187fb66f0e79b787ee634da95b20e4769da7bb92268df8a2677812f3e0d9 not found: ID does not exist"
Oct 30 03:28:01.194822 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.194789    4377 scope.go:122] "RemoveContainer" containerID="9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254"
Oct 30 03:28:01.195027 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:28:01.195013    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254\": container with ID starting with 9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254 not found: ID does not exist" containerID="9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254"
Oct 30 03:28:01.195070 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.195029    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254"} err="failed to get container status \"9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254\": rpc error: code = NotFound desc = could not find container \"9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254\": container with ID starting with 9f1042cfb01704d6000aee6b8e49f9c87ccedc94ac7384cd0026028ac1f8f254 not found: ID does not exist"
Oct 30 03:28:01.223425 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.223391    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"]
Oct 30 03:28:01.236406 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:01.236381    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v1-predictor-c58d788b6-k4d26"]
Oct 30 03:28:02.521204 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.521169    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" path="/var/lib/kubelet/pods/1fe2333f-ba43-4ac6-8a9c-d2016371e28f/volumes"
Oct 30 03:28:02.565636 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.565602    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"]
Oct 30 03:28:02.565944 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.565931    4377 cpu_manager.go:414] "containerMap: removing container" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container"
Oct 30 03:28:02.565990 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.565946    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container"
Oct 30 03:28:02.565990 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.565957    4377 cpu_manager.go:414] "containerMap: removing container" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kube-rbac-proxy"
Oct 30 03:28:02.565990 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.565962    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kube-rbac-proxy"
Oct 30 03:28:02.565990 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.565976    4377 cpu_manager.go:414] "containerMap: removing container" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="storage-initializer"
Oct 30 03:28:02.565990 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.565984    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="storage-initializer"
Oct 30 03:28:02.566146 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.566031    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kserve-container"
Oct 30 03:28:02.566146 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.566040    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe2333f-ba43-4ac6-8a9c-d2016371e28f" containerName="kube-rbac-proxy"
Oct 30 03:28:02.590172 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.590148    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.593703 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.593677    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"]
Oct 30 03:28:02.605887 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.605861    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:28:02.606022 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.605977    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Oct 30 03:28:02.606022 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.605980    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:28:02.618203 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.618179    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:28:02.618398 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.618383    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:28:02.618460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.618419    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:28:02.618460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.618452    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Oct 30 03:28:02.706008 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.705977    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.706230 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.706021    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9rmd\" (UniqueName: \"kubernetes.io/projected/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-api-access-j9rmd\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.706230 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.706049    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f765b60e-554c-4c4d-bb51-cf737ef0fce8-proxy-tls\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.706230 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.706097    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.706230 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.706133    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.806963 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.806934    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.807105 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.806979    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9rmd\" (UniqueName: \"kubernetes.io/projected/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-api-access-j9rmd\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.807105 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.807001    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f765b60e-554c-4c4d-bb51-cf737ef0fce8-proxy-tls\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.807105 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.807019    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.807105 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.807037    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.807422 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.807398    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.807707 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.807687    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.807794 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.807760    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.809400 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.809384    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f765b60e-554c-4c4d-bb51-cf737ef0fce8-proxy-tls\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.830635 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.830616    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9rmd\" (UniqueName: \"kubernetes.io/projected/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-api-access-j9rmd\") pod \"isvc-autogluon-v2-predictor-5f94974c6-m976b\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:02.948869 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:02.948838    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:03.284348 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:03.284323    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"]
Oct 30 03:28:03.286853 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:28:03.286822    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf765b60e_554c_4c4d_bb51_cf737ef0fce8.slice/crio-7baaa37463e2a9e0789bafbf79e3d7af8aa8a3ac9b99a12fe32633b588fc5d9d WatchSource:0}: Error finding container 7baaa37463e2a9e0789bafbf79e3d7af8aa8a3ac9b99a12fe32633b588fc5d9d: Status 404 returned error can't find the container with id 7baaa37463e2a9e0789bafbf79e3d7af8aa8a3ac9b99a12fe32633b588fc5d9d
Oct 30 03:28:04.184062 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:04.184028    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerStarted","Data":"b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83"}
Oct 30 03:28:04.184593 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:04.184094    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerStarted","Data":"7baaa37463e2a9e0789bafbf79e3d7af8aa8a3ac9b99a12fe32633b588fc5d9d"}
Oct 30 03:28:10.258988 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:28:10.258946    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:28:24.263844 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:24.263808    4377 generic.go:356] "Generic (PLEG): container finished" podID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerID="b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83" exitCode=0
Oct 30 03:28:24.264253 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:24.263886    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerDied","Data":"b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83"}
Oct 30 03:28:25.270033 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:25.269992    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerStarted","Data":"1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2"}
Oct 30 03:28:25.270557 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:25.270064    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerStarted","Data":"6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769"}
Oct 30 03:28:25.270557 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:25.270275    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:25.329858 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:25.329801    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" podStartSLOduration=2.955577415 podStartE2EDuration="23.329782088s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.374204673s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:28:25.328859766 +0000 UTC m=+2004.921188572" watchObservedRunningTime="2026-10-30 03:28:25.329782088 +0000 UTC m=+2004.922110896"
Oct 30 03:28:26.274192 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:26.274156    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:32.285034 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:28:32.284997    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:28:40.259129 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:28:40.259090    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:29:10.259420 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:29:10.259379    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:29:40.258797 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:29:40.258756    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:30:03.543101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:03.543061    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:30:10.258859 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:30:10.258816    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:30:14.549733 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:14.549689    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.29:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:30:18.893248 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:18.893213    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"]
Oct 30 03:30:18.893727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:18.893586    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kserve-container" containerID="cri-o://6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769" gracePeriod=30
Oct 30 03:30:18.893727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:18.893592    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kube-rbac-proxy" containerID="cri-o://1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2" gracePeriod=30
Oct 30 03:30:19.715188 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:19.715150    4377 generic.go:356] "Generic (PLEG): container finished" podID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerID="1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2" exitCode=0
Oct 30 03:30:19.715364 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:19.715202    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerDied","Data":"1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2"}
Oct 30 03:30:20.341355 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.341332    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:30:20.470079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.470002    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-cabundle-cert\") pod \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") "
Oct 30 03:30:20.470079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.470041    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f765b60e-554c-4c4d-bb51-cf737ef0fce8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f765b60e-554c-4c4d-bb51-cf737ef0fce8-proxy-tls\") pod \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") "
Oct 30 03:30:20.470079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.470069    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-rbac-proxy-sar-config\") pod \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") "
Oct 30 03:30:20.470367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.470105    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-api-access-j9rmd\" (UniqueName: \"kubernetes.io/projected/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-api-access-j9rmd\") pod \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") "
Oct 30 03:30:20.470367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.470143    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kserve-provision-location\") pod \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\" (UID: \"f765b60e-554c-4c4d-bb51-cf737ef0fce8\") "
Oct 30 03:30:20.470496 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.470465    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-cabundle-cert" pod "f765b60e-554c-4c4d-bb51-cf737ef0fce8" (UID: "f765b60e-554c-4c4d-bb51-cf737ef0fce8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:20.470583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.470559    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-rbac-proxy-sar-config" pod "f765b60e-554c-4c4d-bb51-cf737ef0fce8" (UID: "f765b60e-554c-4c4d-bb51-cf737ef0fce8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:30:20.472240 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.472208    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f765b60e-554c-4c4d-bb51-cf737ef0fce8-proxy-tls" pod "f765b60e-554c-4c4d-bb51-cf737ef0fce8" (UID: "f765b60e-554c-4c4d-bb51-cf737ef0fce8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:30:20.472240 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.472216    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-api-access-j9rmd" pod "f765b60e-554c-4c4d-bb51-cf737ef0fce8" (UID: "f765b60e-554c-4c4d-bb51-cf737ef0fce8"). InnerVolumeSpecName "kube-api-access-j9rmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:30:20.474797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.474772    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kserve-provision-location" pod "f765b60e-554c-4c4d-bb51-cf737ef0fce8" (UID: "f765b60e-554c-4c4d-bb51-cf737ef0fce8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:30:20.571263 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.571231    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.571263 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.571262    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.571446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.571273    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f765b60e-554c-4c4d-bb51-cf737ef0fce8-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.571446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.571281    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.571446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.571290    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9rmd\" (UniqueName: \"kubernetes.io/projected/f765b60e-554c-4c4d-bb51-cf737ef0fce8-kube-api-access-j9rmd\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:30:20.721162 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.721076    4377 generic.go:356] "Generic (PLEG): container finished" podID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerID="6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769" exitCode=0
Oct 30 03:30:20.721162 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.721126    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerDied","Data":"6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769"}
Oct 30 03:30:20.721345 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.721182    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b" event={"ID":"f765b60e-554c-4c4d-bb51-cf737ef0fce8","Type":"ContainerDied","Data":"7baaa37463e2a9e0789bafbf79e3d7af8aa8a3ac9b99a12fe32633b588fc5d9d"}
Oct 30 03:30:20.721345 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.721210    4377 scope.go:122] "RemoveContainer" containerID="1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2"
Oct 30 03:30:20.721345 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.721133    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"
Oct 30 03:30:20.729061 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.729043    4377 scope.go:122] "RemoveContainer" containerID="6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769"
Oct 30 03:30:20.736079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.736056    4377 scope.go:122] "RemoveContainer" containerID="b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83"
Oct 30 03:30:20.743054 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.743038    4377 scope.go:122] "RemoveContainer" containerID="1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2"
Oct 30 03:30:20.743327 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:30:20.743307    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2\": container with ID starting with 1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2 not found: ID does not exist" containerID="1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2"
Oct 30 03:30:20.743369 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.743340    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2"} err="failed to get container status \"1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2\": rpc error: code = NotFound desc = could not find container \"1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2\": container with ID starting with 1d9f539bc27421c1f9f6bfd79dfd7abe1f9287c88ec60ebde109f669414713c2 not found: ID does not exist"
Oct 30 03:30:20.743369 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.743355    4377 scope.go:122] "RemoveContainer" containerID="6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769"
Oct 30 03:30:20.743603 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:30:20.743584    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769\": container with ID starting with 6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769 not found: ID does not exist" containerID="6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769"
Oct 30 03:30:20.743658 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.743610    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769"} err="failed to get container status \"6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769\": rpc error: code = NotFound desc = could not find container \"6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769\": container with ID starting with 6b081490db9765d5354a8837e2550bbee7a614f5814a6e59467c4eeb28381769 not found: ID does not exist"
Oct 30 03:30:20.743658 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.743627    4377 scope.go:122] "RemoveContainer" containerID="b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83"
Oct 30 03:30:20.743850 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:30:20.743834    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83\": container with ID starting with b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83 not found: ID does not exist" containerID="b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83"
Oct 30 03:30:20.743889 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.743856    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83"} err="failed to get container status \"b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83\": rpc error: code = NotFound desc = could not find container \"b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83\": container with ID starting with b37b4f432988022ab6fd08a4cc6b0fe3fcf89c1cd5fac44e370955e93e7e2a83 not found: ID does not exist"
Oct 30 03:30:20.773949 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.773926    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"]
Oct 30 03:30:20.786332 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:20.786307    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-predictor-5f94974c6-m976b"]
Oct 30 03:30:22.463957 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.463926    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"]
Oct 30 03:30:22.464367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464250    4377 cpu_manager.go:414] "containerMap: removing container" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kube-rbac-proxy"
Oct 30 03:30:22.464367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464262    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kube-rbac-proxy"
Oct 30 03:30:22.464367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464286    4377 cpu_manager.go:414] "containerMap: removing container" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kserve-container"
Oct 30 03:30:22.464367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464293    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kserve-container"
Oct 30 03:30:22.464367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464314    4377 cpu_manager.go:414] "containerMap: removing container" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="storage-initializer"
Oct 30 03:30:22.464367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464323    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="storage-initializer"
Oct 30 03:30:22.464625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464394    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kube-rbac-proxy"
Oct 30 03:30:22.464625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.464405    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" containerName="kserve-container"
Oct 30 03:30:22.467567 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.467547    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.483495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.483471    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:30:22.495095 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.495064    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Oct 30 03:30:22.495095 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.495071    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:30:22.495302 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.495132    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:30:22.495302 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.495136    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:30:22.495302 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.495229    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Oct 30 03:30:22.495400 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.495306    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:30:22.497414 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.497398    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"]
Oct 30 03:30:22.514690 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.514666    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f765b60e-554c-4c4d-bb51-cf737ef0fce8" path="/var/lib/kubelet/pods/f765b60e-554c-4c4d-bb51-cf737ef0fce8/volumes"
Oct 30 03:30:22.588578 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.588550    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdqv\" (UniqueName: \"kubernetes.io/projected/efa1f554-b951-4905-82bc-2395b6c72b26-kube-api-access-ccdqv\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.588727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.588587    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.588727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.588690    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.588727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.588713    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa1f554-b951-4905-82bc-2395b6c72b26-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.588835 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.588737    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efa1f554-b951-4905-82bc-2395b6c72b26-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.689952 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.689911    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdqv\" (UniqueName: \"kubernetes.io/projected/efa1f554-b951-4905-82bc-2395b6c72b26-kube-api-access-ccdqv\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.689952 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.689955    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.690185 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.690027    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.690185 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.690046    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa1f554-b951-4905-82bc-2395b6c72b26-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.690185 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.690067    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efa1f554-b951-4905-82bc-2395b6c72b26-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.690541 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.690496    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa1f554-b951-4905-82bc-2395b6c72b26-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.690825 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.690804    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.690912 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.690894    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.692509 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.692487    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efa1f554-b951-4905-82bc-2395b6c72b26-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.713934 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.713909    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdqv\" (UniqueName: \"kubernetes.io/projected/efa1f554-b951-4905-82bc-2395b6c72b26-kube-api-access-ccdqv\") pod \"isvc-autogluon-v2-variants-predictor-8b494d785-274d2\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:22.782003 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:22.781961    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:23.093845 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:23.093746    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"]
Oct 30 03:30:23.097999 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:30:23.097971    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa1f554_b951_4905_82bc_2395b6c72b26.slice/crio-9bbeb59fc8315d4c32fd0c64acff1696b47fa5bed8694efc7807eb9cbd93746a WatchSource:0}: Error finding container 9bbeb59fc8315d4c32fd0c64acff1696b47fa5bed8694efc7807eb9cbd93746a: Status 404 returned error can't find the container with id 9bbeb59fc8315d4c32fd0c64acff1696b47fa5bed8694efc7807eb9cbd93746a
Oct 30 03:30:23.735537 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:23.735484    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerStarted","Data":"57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657"}
Oct 30 03:30:23.735917 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:23.735552    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerStarted","Data":"9bbeb59fc8315d4c32fd0c64acff1696b47fa5bed8694efc7807eb9cbd93746a"}
Oct 30 03:30:40.258935 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:30:40.258898    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:30:43.813990 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:43.813956    4377 generic.go:356] "Generic (PLEG): container finished" podID="efa1f554-b951-4905-82bc-2395b6c72b26" containerID="57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657" exitCode=0
Oct 30 03:30:43.814432 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:43.814029    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerDied","Data":"57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657"}
Oct 30 03:30:44.820952 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:44.820916    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerStarted","Data":"123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792"}
Oct 30 03:30:44.821380 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:44.820981    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerStarted","Data":"9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44"}
Oct 30 03:30:44.821380 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:44.821181    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:44.884965 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:44.884905    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" podStartSLOduration=2.632757384 podStartE2EDuration="22.88488828s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.252130896s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:30:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:30:44.883973598 +0000 UTC m=+2144.476302406" watchObservedRunningTime="2026-10-30 03:30:44.88488828 +0000 UTC m=+2144.477217087"
Oct 30 03:30:45.825691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:45.825658    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:30:51.835253 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:30:51.835222    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:31:10.258233 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:31:10.258192    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:31:40.258949 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:31:40.258913    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:32:10.258281 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:32:10.258234    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:32:17.543256 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:17.543221    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:32:29.196297 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:29.196264    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"]
Oct 30 03:32:29.196745 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:29.196508    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kserve-container" containerID="cri-o://9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44" gracePeriod=30
Oct 30 03:32:29.196745 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:29.196599    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kube-rbac-proxy" containerID="cri-o://123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792" gracePeriod=30
Oct 30 03:32:30.245220 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.245186    4377 generic.go:356] "Generic (PLEG): container finished" podID="efa1f554-b951-4905-82bc-2395b6c72b26" containerID="123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792" exitCode=0
Oct 30 03:32:30.245624 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.245239    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerDied","Data":"123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792"}
Oct 30 03:32:30.644176 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.644147    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:32:30.661717 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.661687    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa1f554-b951-4905-82bc-2395b6c72b26-kube-api-access-ccdqv\" (UniqueName: \"kubernetes.io/projected/efa1f554-b951-4905-82bc-2395b6c72b26-kube-api-access-ccdqv\") pod \"efa1f554-b951-4905-82bc-2395b6c72b26\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") "
Oct 30 03:32:30.661856 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.661726    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-kube-rbac-proxy-sar-config\") pod \"efa1f554-b951-4905-82bc-2395b6c72b26\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") "
Oct 30 03:32:30.661856 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.661829    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-cabundle-cert\") pod \"efa1f554-b951-4905-82bc-2395b6c72b26\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") "
Oct 30 03:32:30.661972 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.661870    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efa1f554-b951-4905-82bc-2395b6c72b26-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa1f554-b951-4905-82bc-2395b6c72b26-kserve-provision-location\") pod \"efa1f554-b951-4905-82bc-2395b6c72b26\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") "
Oct 30 03:32:30.661972 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.661917    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa1f554-b951-4905-82bc-2395b6c72b26-proxy-tls\" (UniqueName: \"kubernetes.io/secret/efa1f554-b951-4905-82bc-2395b6c72b26-proxy-tls\") pod \"efa1f554-b951-4905-82bc-2395b6c72b26\" (UID: \"efa1f554-b951-4905-82bc-2395b6c72b26\") "
Oct 30 03:32:30.662235 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.662208    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-kube-rbac-proxy-sar-config" pod "efa1f554-b951-4905-82bc-2395b6c72b26" (UID: "efa1f554-b951-4905-82bc-2395b6c72b26"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:30.662311 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.662213    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-cabundle-cert" pod "efa1f554-b951-4905-82bc-2395b6c72b26" (UID: "efa1f554-b951-4905-82bc-2395b6c72b26"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:32:30.663874 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.663844    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa1f554-b951-4905-82bc-2395b6c72b26-kube-api-access-ccdqv" pod "efa1f554-b951-4905-82bc-2395b6c72b26" (UID: "efa1f554-b951-4905-82bc-2395b6c72b26"). InnerVolumeSpecName "kube-api-access-ccdqv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:32:30.664031 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.664006    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa1f554-b951-4905-82bc-2395b6c72b26-proxy-tls" pod "efa1f554-b951-4905-82bc-2395b6c72b26" (UID: "efa1f554-b951-4905-82bc-2395b6c72b26"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:32:30.666791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.666765    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efa1f554-b951-4905-82bc-2395b6c72b26-kserve-provision-location" pod "efa1f554-b951-4905-82bc-2395b6c72b26" (UID: "efa1f554-b951-4905-82bc-2395b6c72b26"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:32:30.762793 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.762764    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:32:30.762793 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.762790    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/efa1f554-b951-4905-82bc-2395b6c72b26-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:32:30.762793 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.762803    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/efa1f554-b951-4905-82bc-2395b6c72b26-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:32:30.763017 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.762812    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdqv\" (UniqueName: \"kubernetes.io/projected/efa1f554-b951-4905-82bc-2395b6c72b26-kube-api-access-ccdqv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:32:30.763017 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:30.762820    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/efa1f554-b951-4905-82bc-2395b6c72b26-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:32:31.250724 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.250688    4377 generic.go:356] "Generic (PLEG): container finished" podID="efa1f554-b951-4905-82bc-2395b6c72b26" containerID="9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44" exitCode=0
Oct 30 03:32:31.251204 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.250735    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerDied","Data":"9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44"}
Oct 30 03:32:31.251204 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.250749    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"
Oct 30 03:32:31.251204 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.250798    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2" event={"ID":"efa1f554-b951-4905-82bc-2395b6c72b26","Type":"ContainerDied","Data":"9bbeb59fc8315d4c32fd0c64acff1696b47fa5bed8694efc7807eb9cbd93746a"}
Oct 30 03:32:31.251204 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.250839    4377 scope.go:122] "RemoveContainer" containerID="123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792"
Oct 30 03:32:31.259096 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.259071    4377 scope.go:122] "RemoveContainer" containerID="9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44"
Oct 30 03:32:31.266259 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.266237    4377 scope.go:122] "RemoveContainer" containerID="57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657"
Oct 30 03:32:31.273238 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.273216    4377 scope.go:122] "RemoveContainer" containerID="123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792"
Oct 30 03:32:31.273488 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:32:31.273466    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792\": container with ID starting with 123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792 not found: ID does not exist" containerID="123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792"
Oct 30 03:32:31.273551 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.273499    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792"} err="failed to get container status \"123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792\": rpc error: code = NotFound desc = could not find container \"123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792\": container with ID starting with 123f65f8b288bc446ed094f7a86720d34dcf121dde8b1e21758186cdbb486792 not found: ID does not exist"
Oct 30 03:32:31.273551 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.273527    4377 scope.go:122] "RemoveContainer" containerID="9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44"
Oct 30 03:32:31.273784 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:32:31.273767    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44\": container with ID starting with 9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44 not found: ID does not exist" containerID="9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44"
Oct 30 03:32:31.273825 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.273790    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44"} err="failed to get container status \"9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44\": rpc error: code = NotFound desc = could not find container \"9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44\": container with ID starting with 9793cce26780aadb157b9e48b31fbae488d272799b88d68496bbe09ee8e64b44 not found: ID does not exist"
Oct 30 03:32:31.273825 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.273806    4377 scope.go:122] "RemoveContainer" containerID="57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657"
Oct 30 03:32:31.274056 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:32:31.274037    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657\": container with ID starting with 57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657 not found: ID does not exist" containerID="57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657"
Oct 30 03:32:31.274110 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.274063    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657"} err="failed to get container status \"57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657\": rpc error: code = NotFound desc = could not find container \"57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657\": container with ID starting with 57f9290bcc20fe1ec558653cd9272bc50c8e4f190a36a2392f138bc5eb98f657 not found: ID does not exist"
Oct 30 03:32:31.317112 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.317079    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"]
Oct 30 03:32:31.329604 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:31.329580    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-variants-predictor-8b494d785-274d2"]
Oct 30 03:32:32.515069 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.515037    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" path="/var/lib/kubelet/pods/efa1f554-b951-4905-82bc-2395b6c72b26/volumes"
Oct 30 03:32:32.853985 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.853951    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"]
Oct 30 03:32:32.854304 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854291    4377 cpu_manager.go:414] "containerMap: removing container" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kserve-container"
Oct 30 03:32:32.854495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854305    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kserve-container"
Oct 30 03:32:32.854495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854325    4377 cpu_manager.go:414] "containerMap: removing container" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="storage-initializer"
Oct 30 03:32:32.854495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854330    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="storage-initializer"
Oct 30 03:32:32.854495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854339    4377 cpu_manager.go:414] "containerMap: removing container" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kube-rbac-proxy"
Oct 30 03:32:32.854495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854347    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kube-rbac-proxy"
Oct 30 03:32:32.854495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854411    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kube-rbac-proxy"
Oct 30 03:32:32.854495 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.854426    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa1f554-b951-4905-82bc-2395b6c72b26" containerName="kserve-container"
Oct 30 03:32:32.859270 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.859249    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.879296 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.879268    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.879435 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.879319    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9tqb\" (UniqueName: \"kubernetes.io/projected/b946ad27-cab8-4a02-ba77-51227aa89351-kube-api-access-k9tqb\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.879490 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.879463    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.879490 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.879486    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b946ad27-cab8-4a02-ba77-51227aa89351-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.879594 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.879586    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b946ad27-cab8-4a02-ba77-51227aa89351-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.888003 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.887958    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:32:32.888003 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.887958    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:32:32.888003 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.887962    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:32:32.888223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.887994    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Oct 30 03:32:32.888223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.888016    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:32:32.888223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.888058    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:32:32.888223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.888131    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Oct 30 03:32:32.889342 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.889323    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"]
Oct 30 03:32:32.980849 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.980810    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.980849 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.980847    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b946ad27-cab8-4a02-ba77-51227aa89351-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.981108 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.980891    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b946ad27-cab8-4a02-ba77-51227aa89351-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.981108 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.980907    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.981108 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.980930    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9tqb\" (UniqueName: \"kubernetes.io/projected/b946ad27-cab8-4a02-ba77-51227aa89351-kube-api-access-k9tqb\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.981349 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.981329    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b946ad27-cab8-4a02-ba77-51227aa89351-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.981637 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.981617    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.981707 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.981657    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:32.983246 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:32.983227    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b946ad27-cab8-4a02-ba77-51227aa89351-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:33.004399 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:33.004374    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9tqb\" (UniqueName: \"kubernetes.io/projected/b946ad27-cab8-4a02-ba77-51227aa89351-kube-api-access-k9tqb\") pod \"isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:33.176227 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:33.176133    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:33.488820 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:33.488667    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"]
Oct 30 03:32:34.264895 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:34.264859    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerStarted","Data":"cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8"}
Oct 30 03:32:34.265371 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:34.264933    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerStarted","Data":"d2e4d14e6851040c505618d3355e55e94841f403c61d1f67b513aed2648b288c"}
Oct 30 03:32:39.286069 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:39.286035    4377 generic.go:356] "Generic (PLEG): container finished" podID="b946ad27-cab8-4a02-ba77-51227aa89351" containerID="cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8" exitCode=0
Oct 30 03:32:39.286479 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:39.286092    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerDied","Data":"cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8"}
Oct 30 03:32:40.259830 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:32:40.259788    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:32:40.292592 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:40.292557    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerStarted","Data":"a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a"}
Oct 30 03:32:40.292969 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:40.292680    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerStarted","Data":"8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed"}
Oct 30 03:32:40.292969 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:40.292905    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:40.339452 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:40.339392    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" podStartSLOduration=3.392438767 podStartE2EDuration="8.339372906s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.946934139s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:32:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:32:40.337626907 +0000 UTC m=+2259.929955713" watchObservedRunningTime="2026-10-30 03:32:40.339372906 +0000 UTC m=+2259.931701713"
Oct 30 03:32:41.297138 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:41.297104    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:32:47.307535 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:32:47.307488    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:33:10.260176 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:33:10.260127    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:33:40.258786 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:33:40.258742    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:34:10.258867 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:34:10.258830    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:34:13.545880 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:13.545850    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:34:24.550707 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:24.550659    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.31:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 03:34:29.217437 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:29.217402    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"]
Oct 30 03:34:29.218011 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:29.217664    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kserve-container" containerID="cri-o://8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed" gracePeriod=30
Oct 30 03:34:29.218011 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:29.217734    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kube-rbac-proxy" containerID="cri-o://a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a" gracePeriod=30
Oct 30 03:34:29.722600 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:29.722566    4377 generic.go:356] "Generic (PLEG): container finished" podID="b946ad27-cab8-4a02-ba77-51227aa89351" containerID="a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a" exitCode=0
Oct 30 03:34:29.722772 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:29.722642    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerDied","Data":"a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a"}
Oct 30 03:34:30.659780 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.659758    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:34:30.728925 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.728890    4377 generic.go:356] "Generic (PLEG): container finished" podID="b946ad27-cab8-4a02-ba77-51227aa89351" containerID="8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed" exitCode=0
Oct 30 03:34:30.729101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.728952    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"
Oct 30 03:34:30.729101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.728967    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerDied","Data":"8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed"}
Oct 30 03:34:30.729101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.729044    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4" event={"ID":"b946ad27-cab8-4a02-ba77-51227aa89351","Type":"ContainerDied","Data":"d2e4d14e6851040c505618d3355e55e94841f403c61d1f67b513aed2648b288c"}
Oct 30 03:34:30.729101 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.729081    4377 scope.go:122] "RemoveContainer" containerID="a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a"
Oct 30 03:34:30.738643 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.738619    4377 scope.go:122] "RemoveContainer" containerID="8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed"
Oct 30 03:34:30.746546 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.746513    4377 scope.go:122] "RemoveContainer" containerID="cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8"
Oct 30 03:34:30.753837 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.753810    4377 scope.go:122] "RemoveContainer" containerID="a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a"
Oct 30 03:34:30.754112 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:34:30.754087    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a\": container with ID starting with a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a not found: ID does not exist" containerID="a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a"
Oct 30 03:34:30.754209 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.754120    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a"} err="failed to get container status \"a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a\": rpc error: code = NotFound desc = could not find container \"a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a\": container with ID starting with a88e21cfd9d99c8f78e71b65e43af30a5ef76372e89011a84f58b49848dcb61a not found: ID does not exist"
Oct 30 03:34:30.754209 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.754137    4377 scope.go:122] "RemoveContainer" containerID="8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed"
Oct 30 03:34:30.754443 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:34:30.754417    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed\": container with ID starting with 8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed not found: ID does not exist" containerID="8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed"
Oct 30 03:34:30.754492 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.754449    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed"} err="failed to get container status \"8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed\": rpc error: code = NotFound desc = could not find container \"8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed\": container with ID starting with 8a651bb99b1647b2ba6bdb37e8c35b5f24af2da9bb0653a6cd3bcc6bf8ff24ed not found: ID does not exist"
Oct 30 03:34:30.754492 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.754464    4377 scope.go:122] "RemoveContainer" containerID="cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8"
Oct 30 03:34:30.754690 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:34:30.754673    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8\": container with ID starting with cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8 not found: ID does not exist" containerID="cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8"
Oct 30 03:34:30.754734 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.754693    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8"} err="failed to get container status \"cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8\": rpc error: code = NotFound desc = could not find container \"cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8\": container with ID starting with cb1e8128a5e5c9f5f85f46fe8890d9297ef6a7fd14da5b89b02950929f3937a8 not found: ID does not exist"
Oct 30 03:34:30.755975 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.755959    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-cabundle-cert\") pod \"b946ad27-cab8-4a02-ba77-51227aa89351\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") "
Oct 30 03:34:30.756037 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.756002    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b946ad27-cab8-4a02-ba77-51227aa89351-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b946ad27-cab8-4a02-ba77-51227aa89351-kserve-provision-location\") pod \"b946ad27-cab8-4a02-ba77-51227aa89351\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") "
Oct 30 03:34:30.756194 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.756176    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b946ad27-cab8-4a02-ba77-51227aa89351-kube-api-access-k9tqb\" (UniqueName: \"kubernetes.io/projected/b946ad27-cab8-4a02-ba77-51227aa89351-kube-api-access-k9tqb\") pod \"b946ad27-cab8-4a02-ba77-51227aa89351\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") "
Oct 30 03:34:30.756246 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.756208    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b946ad27-cab8-4a02-ba77-51227aa89351-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b946ad27-cab8-4a02-ba77-51227aa89351-proxy-tls\") pod \"b946ad27-cab8-4a02-ba77-51227aa89351\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") "
Oct 30 03:34:30.756246 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.756236    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-kube-rbac-proxy-sar-config\") pod \"b946ad27-cab8-4a02-ba77-51227aa89351\" (UID: \"b946ad27-cab8-4a02-ba77-51227aa89351\") "
Oct 30 03:34:30.756318 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.756294    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-cabundle-cert" pod "b946ad27-cab8-4a02-ba77-51227aa89351" (UID: "b946ad27-cab8-4a02-ba77-51227aa89351"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:30.756615 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.756590    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-kube-rbac-proxy-sar-config" pod "b946ad27-cab8-4a02-ba77-51227aa89351" (UID: "b946ad27-cab8-4a02-ba77-51227aa89351"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:34:30.756673 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.756616    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:34:30.758228 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.758206    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b946ad27-cab8-4a02-ba77-51227aa89351-kube-api-access-k9tqb" pod "b946ad27-cab8-4a02-ba77-51227aa89351" (UID: "b946ad27-cab8-4a02-ba77-51227aa89351"). InnerVolumeSpecName "kube-api-access-k9tqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:34:30.758325 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.758240    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b946ad27-cab8-4a02-ba77-51227aa89351-proxy-tls" pod "b946ad27-cab8-4a02-ba77-51227aa89351" (UID: "b946ad27-cab8-4a02-ba77-51227aa89351"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:34:30.760460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.760406    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b946ad27-cab8-4a02-ba77-51227aa89351-kserve-provision-location" pod "b946ad27-cab8-4a02-ba77-51227aa89351" (UID: "b946ad27-cab8-4a02-ba77-51227aa89351"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:34:30.857189 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.857156    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9tqb\" (UniqueName: \"kubernetes.io/projected/b946ad27-cab8-4a02-ba77-51227aa89351-kube-api-access-k9tqb\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:34:30.857189 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.857187    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b946ad27-cab8-4a02-ba77-51227aa89351-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:34:30.857189 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.857198    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b946ad27-cab8-4a02-ba77-51227aa89351-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:34:30.857433 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:30.857207    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b946ad27-cab8-4a02-ba77-51227aa89351-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:34:31.075041 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:31.075010    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"]
Oct 30 03:34:31.087562 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:31.087514    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-v2-noslash-predictor-5cd8dbd6c4-7kcg4"]
Oct 30 03:34:32.514735 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.514703    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" path="/var/lib/kubelet/pods/b946ad27-cab8-4a02-ba77-51227aa89351/volumes"
Oct 30 03:34:32.893223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893187    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"]
Oct 30 03:34:32.893540 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893511    4377 cpu_manager.go:414] "containerMap: removing container" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="storage-initializer"
Oct 30 03:34:32.893605 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893539    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="storage-initializer"
Oct 30 03:34:32.893605 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893565    4377 cpu_manager.go:414] "containerMap: removing container" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kserve-container"
Oct 30 03:34:32.893605 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893570    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kserve-container"
Oct 30 03:34:32.893605 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893582    4377 cpu_manager.go:414] "containerMap: removing container" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kube-rbac-proxy"
Oct 30 03:34:32.893605 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893587    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kube-rbac-proxy"
Oct 30 03:34:32.893772 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893638    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kserve-container"
Oct 30 03:34:32.893772 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.893647    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="b946ad27-cab8-4a02-ba77-51227aa89351" containerName="kube-rbac-proxy"
Oct 30 03:34:32.898617 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.898599    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:32.914080 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.914049    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Oct 30 03:34:32.926445 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.926422    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:34:32.926593 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.926428    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Oct 30 03:34:32.926665 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.926457    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:34:32.926665 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.926461    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:34:32.926665 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.926506    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:34:32.926665 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.926541    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:34:32.928012 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.927991    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"]
Oct 30 03:34:32.973141 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.973108    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-api-access-tlx8d\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:32.973283 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.973151    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:32.973283 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.973181    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:32.973283 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.973254    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c59263ff-17aa-411e-9a40-ad5b331b637b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:32.973386 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:32.973307    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c59263ff-17aa-411e-9a40-ad5b331b637b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.074256 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.074209    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-api-access-tlx8d\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.074465 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.074269    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.074465 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.074301    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.074465 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.074334    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c59263ff-17aa-411e-9a40-ad5b331b637b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.074465 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.074389    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c59263ff-17aa-411e-9a40-ad5b331b637b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.074819 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.074797    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c59263ff-17aa-411e-9a40-ad5b331b637b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.074945 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.074924    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.075060 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.075044    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.076723 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.076703    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c59263ff-17aa-411e-9a40-ad5b331b637b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.098534 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.098495    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-api-access-tlx8d\") pod \"isvc-autogluon-ts-v1-predictor-749544699c-nwf7l\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.216685 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.216596    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:33.531168 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.531141    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"]
Oct 30 03:34:33.533100 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:34:33.533070    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59263ff_17aa_411e_9a40_ad5b331b637b.slice/crio-4b77d0d75bcf25fe0aacc94634daca172868c4a3c03b3984161f4c71f2cd4a09 WatchSource:0}: Error finding container 4b77d0d75bcf25fe0aacc94634daca172868c4a3c03b3984161f4c71f2cd4a09: Status 404 returned error can't find the container with id 4b77d0d75bcf25fe0aacc94634daca172868c4a3c03b3984161f4c71f2cd4a09
Oct 30 03:34:33.743674 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.743637    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerStarted","Data":"187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f"}
Oct 30 03:34:33.743855 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:33.743706    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerStarted","Data":"4b77d0d75bcf25fe0aacc94634daca172868c4a3c03b3984161f4c71f2cd4a09"}
Oct 30 03:34:40.258644 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:34:40.258605    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:34:42.779086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:42.779053    4377 generic.go:356] "Generic (PLEG): container finished" podID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerID="187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f" exitCode=0
Oct 30 03:34:42.779496 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:42.779127    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerDied","Data":"187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f"}
Oct 30 03:34:43.785705 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:43.785625    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerStarted","Data":"2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95"}
Oct 30 03:34:43.785705 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:43.785684    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerStarted","Data":"a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a"}
Oct 30 03:34:43.786172 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:43.785886    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:43.846487 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:43.846426    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" podStartSLOduration=2.979815955 podStartE2EDuration="11.846411577s" totalImagesPullingTime="0s" totalInitContainerRuntime="8.866595622s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:34:43.844827347 +0000 UTC m=+2383.437156153" watchObservedRunningTime="2026-10-30 03:34:43.846411577 +0000 UTC m=+2383.438740447"
Oct 30 03:34:44.789319 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:44.789282    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:44.790652 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:44.790626    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 03:34:45.792975 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:45.792927    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 03:34:50.798223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:50.798194    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:34:50.798865 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:34:50.798840    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Oct 30 03:35:00.800288 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:00.800265    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:35:10.261024 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:10.260917    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:35:15.606544 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:15.606494    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"]
Oct 30 03:35:15.607141 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:15.606821    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kserve-container" containerID="cri-o://a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a" gracePeriod=30
Oct 30 03:35:15.607759 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:15.607694    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kube-rbac-proxy" containerID="cri-o://2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95" gracePeriod=30
Oct 30 03:35:15.793531 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:15.793481    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Oct 30 03:35:15.922418 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:15.922317    4377 generic.go:356] "Generic (PLEG): container finished" podID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerID="2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95" exitCode=0
Oct 30 03:35:15.922418 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:15.922377    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerDied","Data":"2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95"}
Oct 30 03:35:17.354704 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.354677    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:35:17.437422 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.437344    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-rbac-proxy-sar-config\") pod \"c59263ff-17aa-411e-9a40-ad5b331b637b\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") "
Oct 30 03:35:17.437422 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.437381    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c59263ff-17aa-411e-9a40-ad5b331b637b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c59263ff-17aa-411e-9a40-ad5b331b637b-proxy-tls\") pod \"c59263ff-17aa-411e-9a40-ad5b331b637b\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") "
Oct 30 03:35:17.437422 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.437411    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-api-access-tlx8d\") pod \"c59263ff-17aa-411e-9a40-ad5b331b637b\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") "
Oct 30 03:35:17.437728 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.437434    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-cabundle-cert\") pod \"c59263ff-17aa-411e-9a40-ad5b331b637b\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") "
Oct 30 03:35:17.437728 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.437469    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c59263ff-17aa-411e-9a40-ad5b331b637b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c59263ff-17aa-411e-9a40-ad5b331b637b-kserve-provision-location\") pod \"c59263ff-17aa-411e-9a40-ad5b331b637b\" (UID: \"c59263ff-17aa-411e-9a40-ad5b331b637b\") "
Oct 30 03:35:17.437845 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.437804    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-rbac-proxy-sar-config" pod "c59263ff-17aa-411e-9a40-ad5b331b637b" (UID: "c59263ff-17aa-411e-9a40-ad5b331b637b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:17.437920 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.437891    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-cabundle-cert" pod "c59263ff-17aa-411e-9a40-ad5b331b637b" (UID: "c59263ff-17aa-411e-9a40-ad5b331b637b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:17.438541 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.438504    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59263ff-17aa-411e-9a40-ad5b331b637b-kserve-provision-location" pod "c59263ff-17aa-411e-9a40-ad5b331b637b" (UID: "c59263ff-17aa-411e-9a40-ad5b331b637b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:17.439459 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.439442    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-api-access-tlx8d" pod "c59263ff-17aa-411e-9a40-ad5b331b637b" (UID: "c59263ff-17aa-411e-9a40-ad5b331b637b"). InnerVolumeSpecName "kube-api-access-tlx8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:35:17.439613 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.439589    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c59263ff-17aa-411e-9a40-ad5b331b637b-proxy-tls" pod "c59263ff-17aa-411e-9a40-ad5b331b637b" (UID: "c59263ff-17aa-411e-9a40-ad5b331b637b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:17.538481 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.538442    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:17.538481 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.538472    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c59263ff-17aa-411e-9a40-ad5b331b637b-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:17.538481 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.538482    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlx8d\" (UniqueName: \"kubernetes.io/projected/c59263ff-17aa-411e-9a40-ad5b331b637b-kube-api-access-tlx8d\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:17.538481 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.538490    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c59263ff-17aa-411e-9a40-ad5b331b637b-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:17.538762 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.538499    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c59263ff-17aa-411e-9a40-ad5b331b637b-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:17.932658 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.932620    4377 generic.go:356] "Generic (PLEG): container finished" podID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerID="a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a" exitCode=0
Oct 30 03:35:17.932817 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.932688    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"
Oct 30 03:35:17.932817 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.932694    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerDied","Data":"a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a"}
Oct 30 03:35:17.932817 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.932757    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l" event={"ID":"c59263ff-17aa-411e-9a40-ad5b331b637b","Type":"ContainerDied","Data":"4b77d0d75bcf25fe0aacc94634daca172868c4a3c03b3984161f4c71f2cd4a09"}
Oct 30 03:35:17.932817 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.932783    4377 scope.go:122] "RemoveContainer" containerID="2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95"
Oct 30 03:35:17.941084 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.941068    4377 scope.go:122] "RemoveContainer" containerID="a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a"
Oct 30 03:35:17.950347 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.950327    4377 scope.go:122] "RemoveContainer" containerID="187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f"
Oct 30 03:35:17.957919 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.957896    4377 scope.go:122] "RemoveContainer" containerID="2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95"
Oct 30 03:35:17.958174 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:17.958153    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95\": container with ID starting with 2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95 not found: ID does not exist" containerID="2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95"
Oct 30 03:35:17.958253 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.958187    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95"} err="failed to get container status \"2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95\": rpc error: code = NotFound desc = could not find container \"2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95\": container with ID starting with 2313ec914120c61433ab302bc8a14cb62d39c93b07413570352533316f257e95 not found: ID does not exist"
Oct 30 03:35:17.958253 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.958213    4377 scope.go:122] "RemoveContainer" containerID="a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a"
Oct 30 03:35:17.958461 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:17.958442    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a\": container with ID starting with a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a not found: ID does not exist" containerID="a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a"
Oct 30 03:35:17.958504 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.958468    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a"} err="failed to get container status \"a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a\": rpc error: code = NotFound desc = could not find container \"a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a\": container with ID starting with a51df05c49c72b2bd718a1acf53d08ec7d6df930cad8530071016067a0e26e8a not found: ID does not exist"
Oct 30 03:35:17.958504 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.958485    4377 scope.go:122] "RemoveContainer" containerID="187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f"
Oct 30 03:35:17.958720 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:17.958702    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f\": container with ID starting with 187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f not found: ID does not exist" containerID="187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f"
Oct 30 03:35:17.958771 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.958726    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f"} err="failed to get container status \"187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f\": rpc error: code = NotFound desc = could not find container \"187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f\": container with ID starting with 187862a53edec4c60f7cdbf143e71795deab998e07062d7f748d03a0cc113b4f not found: ID does not exist"
Oct 30 03:35:17.998319 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:17.998289    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"]
Oct 30 03:35:18.010758 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:18.010726    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-predictor-749544699c-nwf7l"]
Oct 30 03:35:18.514620 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:18.514590    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" path="/var/lib/kubelet/pods/c59263ff-17aa-411e-9a40-ad5b331b637b/volumes"
Oct 30 03:35:19.253293 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253260    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"]
Oct 30 03:35:19.253590 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253575    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kserve-container"
Oct 30 03:35:19.253645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253590    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kserve-container"
Oct 30 03:35:19.253645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253604    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="storage-initializer"
Oct 30 03:35:19.253645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253609    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="storage-initializer"
Oct 30 03:35:19.253645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253625    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kube-rbac-proxy"
Oct 30 03:35:19.253645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253630    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kube-rbac-proxy"
Oct 30 03:35:19.253793 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253695    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kserve-container"
Oct 30 03:35:19.253793 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.253705    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59263ff-17aa-411e-9a40-ad5b331b637b" containerName="kube-rbac-proxy"
Oct 30 03:35:19.258437 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.258419    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.278476 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.278446    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"]
Oct 30 03:35:19.279845 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.279815    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:35:19.289551 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.289514    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Oct 30 03:35:19.289672 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.289563    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:35:19.289672 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.289596    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:35:19.289931 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.289908    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:35:19.289931 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.289933    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:35:19.290067 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.289986    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Oct 30 03:35:19.352430 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.352393    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.352660 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.352443    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.352660 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.352567    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.352660 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.352615    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6vr2\" (UniqueName: \"kubernetes.io/projected/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-api-access-r6vr2\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.352785 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.352665    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4714f6d8-3f8a-485b-9f6b-23736a0204f3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.453405 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.453374    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.453604 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.453431    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.453604 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.453493    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.453733 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.453638    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6vr2\" (UniqueName: \"kubernetes.io/projected/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-api-access-r6vr2\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.453733 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.453674    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4714f6d8-3f8a-485b-9f6b-23736a0204f3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.453861 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.453842    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.454193 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.454172    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.454193 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.454183    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.456005 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.455988    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4714f6d8-3f8a-485b-9f6b-23736a0204f3-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.477187 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.477156    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6vr2\" (UniqueName: \"kubernetes.io/projected/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-api-access-r6vr2\") pod \"isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") " pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.575249 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.575212    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:19.886736 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.886568    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"]
Oct 30 03:35:19.889415 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:35:19.889385    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4714f6d8_3f8a_485b_9f6b_23736a0204f3.slice/crio-d9ddb98dd61252e15bed11e564bfb3bb5b616f5fa49158f1867f03ea627dae2b WatchSource:0}: Error finding container d9ddb98dd61252e15bed11e564bfb3bb5b616f5fa49158f1867f03ea627dae2b: Status 404 returned error can't find the container with id d9ddb98dd61252e15bed11e564bfb3bb5b616f5fa49158f1867f03ea627dae2b
Oct 30 03:35:19.942622 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:19.942596    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerStarted","Data":"d9ddb98dd61252e15bed11e564bfb3bb5b616f5fa49158f1867f03ea627dae2b"}
Oct 30 03:35:20.947679 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:20.947637    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerStarted","Data":"01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf"}
Oct 30 03:35:24.965820 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:24.965728    4377 generic.go:356] "Generic (PLEG): container finished" podID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerID="01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf" exitCode=0
Oct 30 03:35:24.965820 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:24.965765    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerDied","Data":"01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf"}
Oct 30 03:35:25.973538 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:25.973490    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerStarted","Data":"ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52"}
Oct 30 03:35:25.973943 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:25.973567    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerStarted","Data":"8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d"}
Oct 30 03:35:25.973943 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:25.973757    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:26.032563 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:26.032486    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" podStartSLOduration=2.337608223 podStartE2EDuration="7.032463325s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.694855102s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:35:26.031687306 +0000 UTC m=+2425.624016116" watchObservedRunningTime="2026-10-30 03:35:26.032463325 +0000 UTC m=+2425.624792131"
Oct 30 03:35:26.977709 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:26.977675    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:26.979006 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:26.978978    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 03:35:27.982531 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:27.982486    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 03:35:32.987482 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:32.987449    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:32.988009 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:32.987980    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Oct 30 03:35:40.259186 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:40.259148    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:35:42.989700 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:42.989671    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:54.663078 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:54.663045    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"]
Oct 30 03:35:54.663767 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:54.663335    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kserve-container" containerID="cri-o://8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d" gracePeriod=30
Oct 30 03:35:54.663767 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:54.663346    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kube-rbac-proxy" containerID="cri-o://ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52" gracePeriod=30
Oct 30 03:35:55.101885 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:55.101853    4377 generic.go:356] "Generic (PLEG): container finished" podID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerID="ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52" exitCode=0
Oct 30 03:35:55.101885 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:55.101890    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerDied","Data":"ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52"}
Oct 30 03:35:56.098435 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.098410    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:56.107541 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.107497    4377 generic.go:356] "Generic (PLEG): container finished" podID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerID="8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d" exitCode=0
Oct 30 03:35:56.107669 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.107562    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"
Oct 30 03:35:56.107669 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.107554    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerDied","Data":"8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d"}
Oct 30 03:35:56.107741 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.107684    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn" event={"ID":"4714f6d8-3f8a-485b-9f6b-23736a0204f3","Type":"ContainerDied","Data":"d9ddb98dd61252e15bed11e564bfb3bb5b616f5fa49158f1867f03ea627dae2b"}
Oct 30 03:35:56.107741 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.107710    4377 scope.go:122] "RemoveContainer" containerID="ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52"
Oct 30 03:35:56.115721 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.115698    4377 scope.go:122] "RemoveContainer" containerID="8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d"
Oct 30 03:35:56.123140 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.123116    4377 scope.go:122] "RemoveContainer" containerID="01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf"
Oct 30 03:35:56.130375 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.130349    4377 scope.go:122] "RemoveContainer" containerID="ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52"
Oct 30 03:35:56.130658 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:56.130640    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52\": container with ID starting with ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52 not found: ID does not exist" containerID="ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52"
Oct 30 03:35:56.130712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.130669    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52"} err="failed to get container status \"ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52\": rpc error: code = NotFound desc = could not find container \"ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52\": container with ID starting with ea0f801ad5a094f0bc67a3f4b0d8503f6aa79c524ce332dbe0012515af53fd52 not found: ID does not exist"
Oct 30 03:35:56.130712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.130685    4377 scope.go:122] "RemoveContainer" containerID="8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d"
Oct 30 03:35:56.130921 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:56.130907    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d\": container with ID starting with 8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d not found: ID does not exist" containerID="8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d"
Oct 30 03:35:56.130960 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.130926    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d"} err="failed to get container status \"8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d\": rpc error: code = NotFound desc = could not find container \"8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d\": container with ID starting with 8face3230315c6f5c83543ee17f119f3b2dc17a4441c107d8240ae7e8143ce1d not found: ID does not exist"
Oct 30 03:35:56.130960 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.130938    4377 scope.go:122] "RemoveContainer" containerID="01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf"
Oct 30 03:35:56.131150 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:35:56.131135    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf\": container with ID starting with 01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf not found: ID does not exist" containerID="01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf"
Oct 30 03:35:56.131189 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.131155    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf"} err="failed to get container status \"01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf\": rpc error: code = NotFound desc = could not find container \"01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf\": container with ID starting with 01700e2609d0f58d97257df47a5602a738a81e97b9beec8672b9eb971821d8bf not found: ID does not exist"
Oct 30 03:35:56.269350 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.269266    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4714f6d8-3f8a-485b-9f6b-23736a0204f3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4714f6d8-3f8a-485b-9f6b-23736a0204f3-proxy-tls\") pod \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") "
Oct 30 03:35:56.269350 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.269319    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-rbac-proxy-sar-config\") pod \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") "
Oct 30 03:35:56.269588 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.269353    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-api-access-r6vr2\" (UniqueName: \"kubernetes.io/projected/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-api-access-r6vr2\") pod \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") "
Oct 30 03:35:56.269588 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.269423    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kserve-provision-location\") pod \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") "
Oct 30 03:35:56.269588 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.269447    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-cabundle-cert\") pod \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\" (UID: \"4714f6d8-3f8a-485b-9f6b-23736a0204f3\") "
Oct 30 03:35:56.269862 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.269832    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-rbac-proxy-sar-config" pod "4714f6d8-3f8a-485b-9f6b-23736a0204f3" (UID: "4714f6d8-3f8a-485b-9f6b-23736a0204f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:56.269997 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.269879    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-cabundle-cert" pod "4714f6d8-3f8a-485b-9f6b-23736a0204f3" (UID: "4714f6d8-3f8a-485b-9f6b-23736a0204f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:35:56.270505 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.270482    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kserve-provision-location" pod "4714f6d8-3f8a-485b-9f6b-23736a0204f3" (UID: "4714f6d8-3f8a-485b-9f6b-23736a0204f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:35:56.271419 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.271402    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4714f6d8-3f8a-485b-9f6b-23736a0204f3-proxy-tls" pod "4714f6d8-3f8a-485b-9f6b-23736a0204f3" (UID: "4714f6d8-3f8a-485b-9f6b-23736a0204f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:35:56.271545 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.271512    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-api-access-r6vr2" pod "4714f6d8-3f8a-485b-9f6b-23736a0204f3" (UID: "4714f6d8-3f8a-485b-9f6b-23736a0204f3"). InnerVolumeSpecName "kube-api-access-r6vr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:35:56.370601 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.370556    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:56.370601 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.370596    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:56.370798 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.370629    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4714f6d8-3f8a-485b-9f6b-23736a0204f3-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:56.370798 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.370639    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:56.370798 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.370648    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6vr2\" (UniqueName: \"kubernetes.io/projected/4714f6d8-3f8a-485b-9f6b-23736a0204f3-kube-api-access-r6vr2\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:35:56.452892 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.452860    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"]
Oct 30 03:35:56.465425 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.465398    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-autogluon-ts-v1-noslash-predictor-75dfd6d8b-fmxcn"]
Oct 30 03:35:56.514100 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:35:56.514067    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" path="/var/lib/kubelet/pods/4714f6d8-3f8a-485b-9f6b-23736a0204f3/volumes"
Oct 30 03:36:10.257974 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:36:10.257935    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:36:40.257404 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:36:40.257366    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:37:10.258408 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:37:10.258361    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:37:40.259077 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:37:40.259041    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:38:08.297405 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297363    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"]
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297733    4377 cpu_manager.go:414] "containerMap: removing container" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kserve-container"
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297746    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kserve-container"
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297765    4377 cpu_manager.go:414] "containerMap: removing container" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="storage-initializer"
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297770    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="storage-initializer"
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297776    4377 cpu_manager.go:414] "containerMap: removing container" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kube-rbac-proxy"
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297782    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kube-rbac-proxy"
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297854    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kserve-container"
Oct 30 03:38:08.297921 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.297864    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="4714f6d8-3f8a-485b-9f6b-23736a0204f3" containerName="kube-rbac-proxy"
Oct 30 03:38:08.301083 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.301064    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.320543 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.320490    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:38:08.333022 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.332991    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:38:08.333170 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.333099    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Oct 30 03:38:08.333170 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.333098    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:38:08.334631 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.334603    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"]
Oct 30 03:38:08.344133 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.344111    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kserve-provision-location\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.344231 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.344156    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-proxy-tls\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.344294 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.344263    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxjtj\" (UniqueName: \"kubernetes.io/projected/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-api-access-kxjtj\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.344353 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.344321    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.344420 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.344400    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-cabundle-cert\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.361955 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.361929    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:38:08.362042 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.361987    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-create-predictor-serving-cert\""
Oct 30 03:38:08.362042 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.362027    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:38:08.445502 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.445462    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.445691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.445509    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-cabundle-cert\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.445691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.445551    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kserve-provision-location\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.445691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.445577    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-proxy-tls\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.445691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.445604    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxjtj\" (UniqueName: \"kubernetes.io/projected/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-api-access-kxjtj\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.446022 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.446002    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kserve-provision-location\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.446181 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.446162    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-cabundle-cert\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.446288 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.446268    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.447929 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.447911    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-proxy-tls\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.469580 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.469559    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxjtj\" (UniqueName: \"kubernetes.io/projected/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-api-access-kxjtj\") pod \"isvc-canary-create-predictor-6764465dcc-vwgbg\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") " pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.620498 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.620412    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:08.934501 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:08.934484    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"]
Oct 30 03:38:08.936744 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:38:08.936715    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaa6a88_c22d_4078_93c1_03e1cd9126a9.slice/crio-89f086f3ebd5a56c47a8c2780d66710a5782b2a4a8987ccb4961001ab4080d13 WatchSource:0}: Error finding container 89f086f3ebd5a56c47a8c2780d66710a5782b2a4a8987ccb4961001ab4080d13: Status 404 returned error can't find the container with id 89f086f3ebd5a56c47a8c2780d66710a5782b2a4a8987ccb4961001ab4080d13
Oct 30 03:38:09.613052 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:09.613015    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerStarted","Data":"56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39"}
Oct 30 03:38:09.613426 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:09.613073    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerStarted","Data":"89f086f3ebd5a56c47a8c2780d66710a5782b2a4a8987ccb4961001ab4080d13"}
Oct 30 03:38:10.259294 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:38:10.259252    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:38:13.629735 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:13.629694    4377 generic.go:356] "Generic (PLEG): container finished" podID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerID="56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39" exitCode=0
Oct 30 03:38:13.630133 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:13.629767    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerDied","Data":"56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39"}
Oct 30 03:38:13.630952 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:13.630937    4377 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:38:27.701807 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:27.701771    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerStarted","Data":"d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457"}
Oct 30 03:38:27.702282 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:27.701832    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerStarted","Data":"10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4"}
Oct 30 03:38:27.702282 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:27.702073    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:27.702282 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:27.702125    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:27.703674 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:27.703644    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:38:27.747962 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:27.747910    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podStartSLOduration=2.569365797 podStartE2EDuration="19.747895741s" totalImagesPullingTime="13.497888778s" totalInitContainerRuntime="3.680641166s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:38:27.747038491 +0000 UTC m=+2607.339367296" watchObservedRunningTime="2026-10-30 03:38:27.747895741 +0000 UTC m=+2607.340224547"
Oct 30 03:38:28.706449 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:28.706399    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:38:33.711111 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:33.711081    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:38:33.711630 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:33.711600    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:38:40.259058 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:38:40.259019    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:38:43.711570 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:43.711535    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:38:53.711741 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:38:53.711696    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:39:03.711991 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:03.711952    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:39:10.259164 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:39:10.259123    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:39:13.712258 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:13.712163    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:39:23.712534 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:23.712477    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:39:33.711606 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:33.711569    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:39:40.258442 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:39:40.258369    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:39:43.711748 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:43.711711    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Oct 30 03:39:53.713516 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:53.713485    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:39:58.195083 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:58.195049    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"]
Oct 30 03:39:58.195596 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:58.195272    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container" containerID="cri-o://10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4" gracePeriod=30
Oct 30 03:39:58.195596 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:58.195300    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kube-rbac-proxy" containerID="cri-o://d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457" gracePeriod=30
Oct 30 03:39:58.706828 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:58.706787    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Oct 30 03:39:59.073819 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:59.073785    4377 generic.go:356] "Generic (PLEG): container finished" podID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerID="d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457" exitCode=0
Oct 30 03:39:59.073992 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:39:59.073852    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerDied","Data":"d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457"}
Oct 30 03:40:03.245086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.245057    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:40:03.329313 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.329281    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-api-access-kxjtj\" (UniqueName: \"kubernetes.io/projected/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-api-access-kxjtj\") pod \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") "
Oct 30 03:40:03.329490 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.329389    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-cabundle-cert\") pod \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") "
Oct 30 03:40:03.329490 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.329462    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kserve-provision-location\") pod \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") "
Oct 30 03:40:03.329655 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.329538    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-proxy-tls\") pod \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") "
Oct 30 03:40:03.329655 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.329574    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-rbac-proxy-sar-config\") pod \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\" (UID: \"eaaa6a88-c22d-4078-93c1-03e1cd9126a9\") "
Oct 30 03:40:03.329852 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.329814    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kserve-provision-location" pod "eaaa6a88-c22d-4078-93c1-03e1cd9126a9" (UID: "eaaa6a88-c22d-4078-93c1-03e1cd9126a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:40:03.329852 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.329839    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-cabundle-cert" pod "eaaa6a88-c22d-4078-93c1-03e1cd9126a9" (UID: "eaaa6a88-c22d-4078-93c1-03e1cd9126a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:03.330030 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.330006    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-rbac-proxy-sar-config" pod "eaaa6a88-c22d-4078-93c1-03e1cd9126a9" (UID: "eaaa6a88-c22d-4078-93c1-03e1cd9126a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:40:03.331558 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.331537    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-api-access-kxjtj" pod "eaaa6a88-c22d-4078-93c1-03e1cd9126a9" (UID: "eaaa6a88-c22d-4078-93c1-03e1cd9126a9"). InnerVolumeSpecName "kube-api-access-kxjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:40:03.331633 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.331550    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-proxy-tls" pod "eaaa6a88-c22d-4078-93c1-03e1cd9126a9" (UID: "eaaa6a88-c22d-4078-93c1-03e1cd9126a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:40:03.430132 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.430099    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.430132 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.430128    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.430132 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.430138    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.430337 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.430147    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:40:03.430337 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:03.430156    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxjtj\" (UniqueName: \"kubernetes.io/projected/eaaa6a88-c22d-4078-93c1-03e1cd9126a9-kube-api-access-kxjtj\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:40:04.095568 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.095506    4377 generic.go:356] "Generic (PLEG): container finished" podID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerID="10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4" exitCode=0
Oct 30 03:40:04.095775 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.095593    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"
Oct 30 03:40:04.095775 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.095598    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerDied","Data":"10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4"}
Oct 30 03:40:04.095775 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.095673    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg" event={"ID":"eaaa6a88-c22d-4078-93c1-03e1cd9126a9","Type":"ContainerDied","Data":"89f086f3ebd5a56c47a8c2780d66710a5782b2a4a8987ccb4961001ab4080d13"}
Oct 30 03:40:04.095775 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.095710    4377 scope.go:122] "RemoveContainer" containerID="d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457"
Oct 30 03:40:04.104413 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.104390    4377 scope.go:122] "RemoveContainer" containerID="10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4"
Oct 30 03:40:04.111586 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.111570    4377 scope.go:122] "RemoveContainer" containerID="56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39"
Oct 30 03:40:04.118659 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.118642    4377 scope.go:122] "RemoveContainer" containerID="d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457"
Oct 30 03:40:04.118941 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:40:04.118922    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457\": container with ID starting with d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457 not found: ID does not exist" containerID="d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457"
Oct 30 03:40:04.119000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.118950    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457"} err="failed to get container status \"d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457\": rpc error: code = NotFound desc = could not find container \"d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457\": container with ID starting with d62828fee4e0422ec7e1e2ab7b9acdc43d7fcc1b8b92b6cbbf7d8c806df14457 not found: ID does not exist"
Oct 30 03:40:04.119000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.118967    4377 scope.go:122] "RemoveContainer" containerID="10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4"
Oct 30 03:40:04.119217 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:40:04.119200    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4\": container with ID starting with 10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4 not found: ID does not exist" containerID="10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4"
Oct 30 03:40:04.119269 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.119221    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4"} err="failed to get container status \"10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4\": rpc error: code = NotFound desc = could not find container \"10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4\": container with ID starting with 10e3f0dd49ea4b33715639d86509058222efcaacecd567637107bf1596ce04f4 not found: ID does not exist"
Oct 30 03:40:04.119269 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.119235    4377 scope.go:122] "RemoveContainer" containerID="56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39"
Oct 30 03:40:04.119466 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:40:04.119449    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39\": container with ID starting with 56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39 not found: ID does not exist" containerID="56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39"
Oct 30 03:40:04.119504 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.119473    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39"} err="failed to get container status \"56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39\": rpc error: code = NotFound desc = could not find container \"56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39\": container with ID starting with 56e6d95bf87545d768bff6e09668d70347d0099cf16919318afe37a29af43b39 not found: ID does not exist"
Oct 30 03:40:04.163160 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.163123    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"]
Oct 30 03:40:04.167125 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.167102    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-create-predictor-6764465dcc-vwgbg"]
Oct 30 03:40:04.333414 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333379    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"]
Oct 30 03:40:04.333786 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333722    4377 cpu_manager.go:414] "containerMap: removing container" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container"
Oct 30 03:40:04.333786 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333734    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container"
Oct 30 03:40:04.333786 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333749    4377 cpu_manager.go:414] "containerMap: removing container" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kube-rbac-proxy"
Oct 30 03:40:04.333786 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333761    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kube-rbac-proxy"
Oct 30 03:40:04.333786 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333774    4377 cpu_manager.go:414] "containerMap: removing container" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="storage-initializer"
Oct 30 03:40:04.333786 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333779    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="storage-initializer"
Oct 30 03:40:04.333982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333846    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kserve-container"
Oct 30 03:40:04.333982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.333855    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" containerName="kube-rbac-proxy"
Oct 30 03:40:04.338505 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.338485    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.356840 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.356627    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Oct 30 03:40:04.358077 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.358057    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"]
Oct 30 03:40:04.359361 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.359334    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:40:04.359532 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.359500    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:40:04.359603 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.359563    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:40:04.359669 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.359627    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:40:04.437411 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.437372    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.437647 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.437418    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.437647 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.437575    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.437647 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.437623    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnrc\" (UniqueName: \"kubernetes.io/projected/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-api-access-nfnrc\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.437775 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.437729    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.457373 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.457349    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:40:04.457450 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.457384    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:40:04.513972 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.513940    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaaa6a88-c22d-4078-93c1-03e1cd9126a9" path="/var/lib/kubelet/pods/eaaa6a88-c22d-4078-93c1-03e1cd9126a9/volumes"
Oct 30 03:40:04.538874 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.538845    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.539008 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.538880    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.539008 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.538911    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.539008 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.538943    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnrc\" (UniqueName: \"kubernetes.io/projected/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-api-access-nfnrc\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.539008 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.538988    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.539313 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.539292    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.539600 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.539582    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.539710 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.539693    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.541381 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.541356    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.563340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.563304    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnrc\" (UniqueName: \"kubernetes.io/projected/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-api-access-nfnrc\") pod \"isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") " pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.655617 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.655501    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:04.984166 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:04.984131    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"]
Oct 30 03:40:05.103133 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:05.103098    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerStarted","Data":"5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468"}
Oct 30 03:40:05.103352 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:05.103170    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerStarted","Data":"214f32e051c5e615223b962d73ed35a6e0a662858ca2a83405a41753849ca2dc"}
Oct 30 03:40:06.204447 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.204400    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"]
Oct 30 03:40:06.208133 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.208114    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.223665 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.223638    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 03:40:06.223787 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.223685    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:40:06.223787 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.223696    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:40:06.226167 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.226152    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:40:06.238030 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.238001    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"]
Oct 30 03:40:06.254783 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.254756    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3a3531-ac2c-423b-9326-dae4634d629c-proxy-tls\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.254938 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.254793    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncxtm\" (UniqueName: \"kubernetes.io/projected/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-api-access-ncxtm\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.254938 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.254813    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb3a3531-ac2c-423b-9326-dae4634d629c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.254938 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.254829    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-cabundle-cert\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.255125 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.255095    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.287150 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.287116    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-promote-predictor-serving-cert\""
Oct 30 03:40:06.287297 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.287203    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:40:06.287297 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.287231    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:40:06.355510 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.355471    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.355510 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.355531    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3a3531-ac2c-423b-9326-dae4634d629c-proxy-tls\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.355797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.355571    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncxtm\" (UniqueName: \"kubernetes.io/projected/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-api-access-ncxtm\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.355797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.355597    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb3a3531-ac2c-423b-9326-dae4634d629c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.355797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.355622    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-cabundle-cert\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.356055 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.356024    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb3a3531-ac2c-423b-9326-dae4634d629c-kserve-provision-location\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.356269 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.356249    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.356364 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.356342    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-cabundle-cert\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.357993 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.357975    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3a3531-ac2c-423b-9326-dae4634d629c-proxy-tls\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.379723 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.379692    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncxtm\" (UniqueName: \"kubernetes.io/projected/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-api-access-ncxtm\") pod \"isvc-canary-promote-predictor-5894dcd7f5-l2wtt\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.525732 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.525652    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:06.839411 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:06.839380    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"]
Oct 30 03:40:06.842194 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:40:06.842167    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3a3531_ac2c_423b_9326_dae4634d629c.slice/crio-dda1cd5f9d3c8236286accb28a0578c00923dcced0d5b32166c48c0a1dc9f50f WatchSource:0}: Error finding container dda1cd5f9d3c8236286accb28a0578c00923dcced0d5b32166c48c0a1dc9f50f: Status 404 returned error can't find the container with id dda1cd5f9d3c8236286accb28a0578c00923dcced0d5b32166c48c0a1dc9f50f
Oct 30 03:40:07.115688 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:07.115593    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerStarted","Data":"79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153"}
Oct 30 03:40:07.115688 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:07.115653    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerStarted","Data":"dda1cd5f9d3c8236286accb28a0578c00923dcced0d5b32166c48c0a1dc9f50f"}
Oct 30 03:40:10.129988 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:10.129899    4377 generic.go:356] "Generic (PLEG): container finished" podID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerID="5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468" exitCode=0
Oct 30 03:40:10.129988 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:10.129977    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerDied","Data":"5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468"}
Oct 30 03:40:10.258659 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:40:10.258621    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:40:11.137848 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:11.136766    4377 generic.go:356] "Generic (PLEG): container finished" podID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerID="79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153" exitCode=0
Oct 30 03:40:11.137848 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:11.136860    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerDied","Data":"79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153"}
Oct 30 03:40:12.143876 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:12.143839    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerStarted","Data":"8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6"}
Oct 30 03:40:12.144367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:12.143911    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerStarted","Data":"c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb"}
Oct 30 03:40:12.144367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:12.144116    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:13.149731 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:13.149701    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:13.151004 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:13.150969    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:40:14.155839 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.155800    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerStarted","Data":"7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5"}
Oct 30 03:40:14.156280 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.155863    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerStarted","Data":"0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51"}
Oct 30 03:40:14.156280 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.156060    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:40:14.156368 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.156278    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:14.156446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.156429    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:14.157569 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.157550    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 03:40:14.214539 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.214484    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podStartSLOduration=4.239576097 podStartE2EDuration="8.214466989s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.974890892s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:12.205669598 +0000 UTC m=+2711.797998403" watchObservedRunningTime="2026-10-30 03:40:14.214466989 +0000 UTC m=+2713.806795795"
Oct 30 03:40:14.215080 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:14.215054    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podStartSLOduration=1.7722597119999999 podStartE2EDuration="10.21504625s" totalImagesPullingTime="3.62672501s" totalInitContainerRuntime="4.816061528s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:40:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:40:14.213710833 +0000 UTC m=+2713.806039656" watchObservedRunningTime="2026-10-30 03:40:14.21504625 +0000 UTC m=+2713.807375133"
Oct 30 03:40:15.160102 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:15.160058    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 03:40:19.161025 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:19.160997    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:40:19.161599 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:19.161572    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:40:20.164864 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:20.164835    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:20.165405 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:20.165377    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Oct 30 03:40:29.162549 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:29.162495    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:40:30.166566 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:30.166540    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:40:39.161914 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:39.161860    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:40:40.259080 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:40:40.259040    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:40:45.708345 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:45.708267    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"]
Oct 30 03:40:45.708764 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:45.708579    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kserve-container" containerID="cri-o://0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51" gracePeriod=30
Oct 30 03:40:45.708764 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:45.708598    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy" containerID="cri-o://7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5" gracePeriod=30
Oct 30 03:40:46.296900 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:46.296867    4377 generic.go:356] "Generic (PLEG): container finished" podID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerID="7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5" exitCode=0
Oct 30 03:40:46.297073 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:46.296919    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerDied","Data":"7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5"}
Oct 30 03:40:49.161927 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:49.161881    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:40:50.160269 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:50.160227    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 03:40:55.160402 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:55.160356    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 03:40:59.162154 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:40:59.162117    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:41:00.161093 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:00.161048    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 03:41:00.161308 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:00.161171    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:41:05.161252 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:05.161203    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 03:41:09.161702 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:09.161662    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:41:10.160957 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:10.160916    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 03:41:10.259172 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:10.259120    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:41:15.161285 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:15.161239    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Oct 30 03:41:16.343911 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.343884    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:41:16.429269 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.429234    4377 generic.go:356] "Generic (PLEG): container finished" podID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerID="0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51" exitCode=137
Oct 30 03:41:16.429460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.429291    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"
Oct 30 03:41:16.429460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.429304    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerDied","Data":"0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51"}
Oct 30 03:41:16.429460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.429397    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9" event={"ID":"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf","Type":"ContainerDied","Data":"214f32e051c5e615223b962d73ed35a6e0a662858ca2a83405a41753849ca2dc"}
Oct 30 03:41:16.429460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.429433    4377 scope.go:122] "RemoveContainer" containerID="7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5"
Oct 30 03:41:16.437543 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.437526    4377 scope.go:122] "RemoveContainer" containerID="0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51"
Oct 30 03:41:16.444806 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.444789    4377 scope.go:122] "RemoveContainer" containerID="5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468"
Oct 30 03:41:16.450940 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.450920    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kserve-provision-location\") pod \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") "
Oct 30 03:41:16.451043 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.450967    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-api-access-nfnrc\" (UniqueName: \"kubernetes.io/projected/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-api-access-nfnrc\") pod \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") "
Oct 30 03:41:16.451043 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.451022    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-proxy-tls\") pod \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") "
Oct 30 03:41:16.451186 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.451053    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-rbac-proxy-sar-config\") pod \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") "
Oct 30 03:41:16.451186 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.451171    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-cabundle-cert\") pod \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\" (UID: \"c83a3bd7-a0f7-42c6-9d39-9f9d535193bf\") "
Oct 30 03:41:16.451461 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.451437    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-rbac-proxy-sar-config" pod "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" (UID: "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:16.451732 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.451712    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-cabundle-cert" pod "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" (UID: "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:16.452311 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.452295    4377 scope.go:122] "RemoveContainer" containerID="7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5"
Oct 30 03:41:16.452635 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:16.452612    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5\": container with ID starting with 7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5 not found: ID does not exist" containerID="7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5"
Oct 30 03:41:16.452720 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.452641    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5"} err="failed to get container status \"7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5\": rpc error: code = NotFound desc = could not find container \"7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5\": container with ID starting with 7afee133ff08083f2e95f92c2383856739317e37ef8d75176a24fa5a035d41c5 not found: ID does not exist"
Oct 30 03:41:16.452720 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.452658    4377 scope.go:122] "RemoveContainer" containerID="0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51"
Oct 30 03:41:16.452917 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:16.452899    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51\": container with ID starting with 0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51 not found: ID does not exist" containerID="0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51"
Oct 30 03:41:16.452964 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.452923    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51"} err="failed to get container status \"0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51\": rpc error: code = NotFound desc = could not find container \"0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51\": container with ID starting with 0f44101d9177bd10d818e5abab86d06f4ab748c4bdac0f734033f3aa7bf60f51 not found: ID does not exist"
Oct 30 03:41:16.452964 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.452943    4377 scope.go:122] "RemoveContainer" containerID="5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468"
Oct 30 03:41:16.453208 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:16.453186    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468\": container with ID starting with 5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468 not found: ID does not exist" containerID="5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468"
Oct 30 03:41:16.453266 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.453217    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468"} err="failed to get container status \"5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468\": rpc error: code = NotFound desc = could not find container \"5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468\": container with ID starting with 5032d786fa5de25fd131541954c08143d7ca02a3d1806b5b9e73787bbe2f1468 not found: ID does not exist"
Oct 30 03:41:16.453601 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.453582    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-proxy-tls" pod "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" (UID: "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:16.453705 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.453683    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-api-access-nfnrc" pod "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" (UID: "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf"). InnerVolumeSpecName "kube-api-access-nfnrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:16.461453 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.461432    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kserve-provision-location" pod "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" (UID: "c83a3bd7-a0f7-42c6-9d39-9f9d535193bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:41:16.552067 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.552041    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:16.552067 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.552064    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:16.552244 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.552074    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfnrc\" (UniqueName: \"kubernetes.io/projected/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-api-access-nfnrc\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:16.552244 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.552085    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:16.552244 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.552093    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:16.769433 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.769399    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"]
Oct 30 03:41:16.781893 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:16.781866    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-tensorflow-runtime-predictor-7c9d8bbc9c-rklp9"]
Oct 30 03:41:18.513852 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:18.513819    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" path="/var/lib/kubelet/pods/c83a3bd7-a0f7-42c6-9d39-9f9d535193bf/volumes"
Oct 30 03:41:19.161669 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:19.161631    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:41:26.510347 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:26.510309    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Oct 30 03:41:36.513638 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:36.513609    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:41:40.102946 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:40.102911    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"]
Oct 30 03:41:40.103340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:40.103160    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container" containerID="cri-o://c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb" gracePeriod=30
Oct 30 03:41:40.103340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:40.103195    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kube-rbac-proxy" containerID="cri-o://8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6" gracePeriod=30
Oct 30 03:41:40.257683 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:40.257639    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:41:40.529291 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:40.529265    4377 generic.go:356] "Generic (PLEG): container finished" podID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerID="8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6" exitCode=0
Oct 30 03:41:40.529446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:40.529335    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerDied","Data":"8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6"}
Oct 30 03:41:44.156846 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.156800    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Oct 30 03:41:44.947608 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.947585    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:41:44.984452 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984427    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-api-access-ncxtm\" (UniqueName: \"kubernetes.io/projected/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-api-access-ncxtm\") pod \"cb3a3531-ac2c-423b-9326-dae4634d629c\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") "
Oct 30 03:41:44.984604 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984460    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-rbac-proxy-sar-config\") pod \"cb3a3531-ac2c-423b-9326-dae4634d629c\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") "
Oct 30 03:41:44.984604 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984484    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-cabundle-cert\") pod \"cb3a3531-ac2c-423b-9326-dae4634d629c\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") "
Oct 30 03:41:44.984604 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984570    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3a3531-ac2c-423b-9326-dae4634d629c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb3a3531-ac2c-423b-9326-dae4634d629c-kserve-provision-location\") pod \"cb3a3531-ac2c-423b-9326-dae4634d629c\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") "
Oct 30 03:41:44.984771 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984683    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3a3531-ac2c-423b-9326-dae4634d629c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3a3531-ac2c-423b-9326-dae4634d629c-proxy-tls\") pod \"cb3a3531-ac2c-423b-9326-dae4634d629c\" (UID: \"cb3a3531-ac2c-423b-9326-dae4634d629c\") "
Oct 30 03:41:44.984947 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984917    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-cabundle-cert" pod "cb3a3531-ac2c-423b-9326-dae4634d629c" (UID: "cb3a3531-ac2c-423b-9326-dae4634d629c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:44.984947 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984930    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-rbac-proxy-sar-config" pod "cb3a3531-ac2c-423b-9326-dae4634d629c" (UID: "cb3a3531-ac2c-423b-9326-dae4634d629c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:41:44.985086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.984970    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3a3531-ac2c-423b-9326-dae4634d629c-kserve-provision-location" pod "cb3a3531-ac2c-423b-9326-dae4634d629c" (UID: "cb3a3531-ac2c-423b-9326-dae4634d629c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:41:44.986614 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.986588    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3a3531-ac2c-423b-9326-dae4634d629c-proxy-tls" pod "cb3a3531-ac2c-423b-9326-dae4634d629c" (UID: "cb3a3531-ac2c-423b-9326-dae4634d629c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:41:44.986694 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:44.986634    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-api-access-ncxtm" pod "cb3a3531-ac2c-423b-9326-dae4634d629c" (UID: "cb3a3531-ac2c-423b-9326-dae4634d629c"). InnerVolumeSpecName "kube-api-access-ncxtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:41:45.085431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.085336    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3a3531-ac2c-423b-9326-dae4634d629c-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:45.085431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.085368    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncxtm\" (UniqueName: \"kubernetes.io/projected/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-api-access-ncxtm\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:45.085431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.085384    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:45.085431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.085399    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cb3a3531-ac2c-423b-9326-dae4634d629c-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:45.085431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.085411    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cb3a3531-ac2c-423b-9326-dae4634d629c-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:41:45.550744 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.550711    4377 generic.go:356] "Generic (PLEG): container finished" podID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerID="c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb" exitCode=0
Oct 30 03:41:45.551225 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.550769    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"
Oct 30 03:41:45.551225 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.550799    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerDied","Data":"c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb"}
Oct 30 03:41:45.551225 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.550872    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt" event={"ID":"cb3a3531-ac2c-423b-9326-dae4634d629c","Type":"ContainerDied","Data":"dda1cd5f9d3c8236286accb28a0578c00923dcced0d5b32166c48c0a1dc9f50f"}
Oct 30 03:41:45.551225 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.550915    4377 scope.go:122] "RemoveContainer" containerID="8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6"
Oct 30 03:41:45.559438 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.559423    4377 scope.go:122] "RemoveContainer" containerID="c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb"
Oct 30 03:41:45.566719 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.566700    4377 scope.go:122] "RemoveContainer" containerID="79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153"
Oct 30 03:41:45.574064 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.574049    4377 scope.go:122] "RemoveContainer" containerID="8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6"
Oct 30 03:41:45.574328 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:45.574311    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6\": container with ID starting with 8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6 not found: ID does not exist" containerID="8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6"
Oct 30 03:41:45.574376 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.574338    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6"} err="failed to get container status \"8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6\": rpc error: code = NotFound desc = could not find container \"8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6\": container with ID starting with 8f300c239d56f48f509d70be1047cfa95549c5a4292ebe7080b747a4117528f6 not found: ID does not exist"
Oct 30 03:41:45.574376 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.574354    4377 scope.go:122] "RemoveContainer" containerID="c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb"
Oct 30 03:41:45.574625 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:45.574606    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb\": container with ID starting with c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb not found: ID does not exist" containerID="c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb"
Oct 30 03:41:45.574678 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.574632    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb"} err="failed to get container status \"c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb\": rpc error: code = NotFound desc = could not find container \"c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb\": container with ID starting with c11ac33b6a395dd3e357fa89563ab40b4d8ba5f4da67998c50f780445fa994cb not found: ID does not exist"
Oct 30 03:41:45.574678 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.574648    4377 scope.go:122] "RemoveContainer" containerID="79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153"
Oct 30 03:41:45.574896 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:41:45.574879    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153\": container with ID starting with 79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153 not found: ID does not exist" containerID="79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153"
Oct 30 03:41:45.574935 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.574903    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153"} err="failed to get container status \"79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153\": rpc error: code = NotFound desc = could not find container \"79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153\": container with ID starting with 79d62c0abd42e64210c24966f4617e924912df8b5420e0fbc8d3cbca3acc6153 not found: ID does not exist"
Oct 30 03:41:45.616667 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.616641    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"]
Oct 30 03:41:45.629370 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:45.629346    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-predictor-5894dcd7f5-l2wtt"]
Oct 30 03:41:46.172158 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172120    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"]
Oct 30 03:41:46.172478 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172459    4377 cpu_manager.go:414] "containerMap: removing container" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kube-rbac-proxy"
Oct 30 03:41:46.172583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172481    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kube-rbac-proxy"
Oct 30 03:41:46.172583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172496    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy"
Oct 30 03:41:46.172583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172504    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy"
Oct 30 03:41:46.172583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172548    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="storage-initializer"
Oct 30 03:41:46.172583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172556    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="storage-initializer"
Oct 30 03:41:46.172583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172576    4377 cpu_manager.go:414] "containerMap: removing container" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="storage-initializer"
Oct 30 03:41:46.172583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172583    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="storage-initializer"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172594    4377 cpu_manager.go:414] "containerMap: removing container" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172602    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172615    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kserve-container"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172623    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kserve-container"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172703    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kserve-container"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172712    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kserve-container"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172719    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" containerName="kube-rbac-proxy"
Oct 30 03:41:46.172791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.172724    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83a3bd7-a0f7-42c6-9d39-9f9d535193bf" containerName="kube-rbac-proxy"
Oct 30 03:41:46.177128 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.177111    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.192428 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.192409    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:41:46.194846 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.194816    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.194961 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.194850    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.194961 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.194892    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.194961 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.194901    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Oct 30 03:41:46.194961 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.194916    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796tv\" (UniqueName: \"kubernetes.io/projected/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-api-access-796tv\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.194961 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.194960    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.197063 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.197040    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"]
Oct 30 03:41:46.205300 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.205276    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:41:46.205300 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.205292    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Oct 30 03:41:46.205470 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.205297    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:41:46.293230 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.293199    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:41:46.293399 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.293279    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:41:46.295640 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.295620    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.295689 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.295651    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.295738 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.295687    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.295910 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.295885    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796tv\" (UniqueName: \"kubernetes.io/projected/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-api-access-796tv\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.296014 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.295935    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.296296 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.296277    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.296382 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.296284    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.296382 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.296357    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.298116 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.298094    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.319630 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.319605    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-796tv\" (UniqueName: \"kubernetes.io/projected/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-api-access-796tv\") pod \"isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") " pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.493800 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.493702    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:46.514896 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.514866    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3a3531-ac2c-423b-9326-dae4634d629c" path="/var/lib/kubelet/pods/cb3a3531-ac2c-423b-9326-dae4634d629c/volumes"
Oct 30 03:41:46.805764 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:46.805739    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"]
Oct 30 03:41:46.807998 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:41:46.807958    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3487ecc1_6243_44ed_a13e_2c5ade7ddae7.slice/crio-615cdfe50e0208a8e8b650d19c2b160746158f69ac468ee827058f52723c28f5 WatchSource:0}: Error finding container 615cdfe50e0208a8e8b650d19c2b160746158f69ac468ee827058f52723c28f5: Status 404 returned error can't find the container with id 615cdfe50e0208a8e8b650d19c2b160746158f69ac468ee827058f52723c28f5
Oct 30 03:41:47.562387 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:47.562348    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerStarted","Data":"d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c"}
Oct 30 03:41:47.562569 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:47.562415    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerStarted","Data":"615cdfe50e0208a8e8b650d19c2b160746158f69ac468ee827058f52723c28f5"}
Oct 30 03:41:51.579473 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:51.579441    4377 generic.go:356] "Generic (PLEG): container finished" podID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerID="d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c" exitCode=0
Oct 30 03:41:51.579841 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:51.579506    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerDied","Data":"d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c"}
Oct 30 03:41:52.586255 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:52.586217    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerStarted","Data":"55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86"}
Oct 30 03:41:52.586754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:52.586283    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerStarted","Data":"34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b"}
Oct 30 03:41:52.586754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:52.586576    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:52.586754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:52.586747    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:52.588246 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:52.588217    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:41:52.644555 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:52.644488    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podStartSLOduration=2.841777263 podStartE2EDuration="6.644471309s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.802694046s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:41:52.644240746 +0000 UTC m=+2812.236569553" watchObservedRunningTime="2026-10-30 03:41:52.644471309 +0000 UTC m=+2812.236800118"
Oct 30 03:41:53.590992 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:53.590948    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:41:58.595633 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:58.595603    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:41:58.596205 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:41:58.596174    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:42:08.599079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:42:08.599026    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:42:10.259035 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:42:10.258996    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:42:18.597012 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:42:18.596933    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:42:28.596020 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:42:28.595983    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:42:38.596529 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:42:38.596473    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:42:40.258780 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:42:40.258742    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:42:48.597019 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:42:48.596966    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:42:58.597016 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:42:58.596975    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:42:59.510013 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:42:59.509974    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:43:09.510081 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:09.510040    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:43:10.259033 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:43:10.258996    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:43:19.511216 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:19.511184    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:43:26.740842 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:26.740802    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"]
Oct 30 03:43:26.741251 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:26.741131    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" containerID="cri-o://34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b" gracePeriod=30
Oct 30 03:43:26.741251 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:26.741194    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kube-rbac-proxy" containerID="cri-o://55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86" gracePeriod=30
Oct 30 03:43:26.962668 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:26.962636    4377 generic.go:356] "Generic (PLEG): container finished" podID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerID="55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86" exitCode=0
Oct 30 03:43:26.962846 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:26.962708    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerDied","Data":"55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86"}
Oct 30 03:43:28.591841 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:28.591799    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Oct 30 03:43:29.510183 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:29.510145    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Oct 30 03:43:31.793129 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.793100    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:43:31.877364 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877285    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-rbac-proxy-sar-config\") pod \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") "
Oct 30 03:43:31.877364 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877319    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-cabundle-cert\") pod \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") "
Oct 30 03:43:31.877364 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877351    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-proxy-tls\") pod \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") "
Oct 30 03:43:31.877662 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877387    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kserve-provision-location\") pod \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") "
Oct 30 03:43:31.877662 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877408    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-api-access-796tv\" (UniqueName: \"kubernetes.io/projected/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-api-access-796tv\") pod \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\" (UID: \"3487ecc1-6243-44ed-a13e-2c5ade7ddae7\") "
Oct 30 03:43:31.877783 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877690    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-rbac-proxy-sar-config" pod "3487ecc1-6243-44ed-a13e-2c5ade7ddae7" (UID: "3487ecc1-6243-44ed-a13e-2c5ade7ddae7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:31.877783 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877721    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kserve-provision-location" pod "3487ecc1-6243-44ed-a13e-2c5ade7ddae7" (UID: "3487ecc1-6243-44ed-a13e-2c5ade7ddae7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:43:31.877883 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.877830    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-cabundle-cert" pod "3487ecc1-6243-44ed-a13e-2c5ade7ddae7" (UID: "3487ecc1-6243-44ed-a13e-2c5ade7ddae7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:43:31.879450 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.879429    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-proxy-tls" pod "3487ecc1-6243-44ed-a13e-2c5ade7ddae7" (UID: "3487ecc1-6243-44ed-a13e-2c5ade7ddae7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:43:31.879507 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.879491    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-api-access-796tv" pod "3487ecc1-6243-44ed-a13e-2c5ade7ddae7" (UID: "3487ecc1-6243-44ed-a13e-2c5ade7ddae7"). InnerVolumeSpecName "kube-api-access-796tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:43:31.978699 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.978666    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.978699 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.978696    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.978913 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.978709    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.978913 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.978722    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.978913 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.978735    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796tv\" (UniqueName: \"kubernetes.io/projected/3487ecc1-6243-44ed-a13e-2c5ade7ddae7-kube-api-access-796tv\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:43:31.984122 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.984092    4377 generic.go:356] "Generic (PLEG): container finished" podID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerID="34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b" exitCode=0
Oct 30 03:43:31.984272 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.984133    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerDied","Data":"34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b"}
Oct 30 03:43:31.984272 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.984167    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"
Oct 30 03:43:31.984272 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.984179    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4" event={"ID":"3487ecc1-6243-44ed-a13e-2c5ade7ddae7","Type":"ContainerDied","Data":"615cdfe50e0208a8e8b650d19c2b160746158f69ac468ee827058f52723c28f5"}
Oct 30 03:43:31.984272 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.984203    4377 scope.go:122] "RemoveContainer" containerID="55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86"
Oct 30 03:43:31.993874 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:31.993855    4377 scope.go:122] "RemoveContainer" containerID="34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b"
Oct 30 03:43:32.002490 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.002468    4377 scope.go:122] "RemoveContainer" containerID="d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c"
Oct 30 03:43:32.009745 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.009716    4377 scope.go:122] "RemoveContainer" containerID="55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86"
Oct 30 03:43:32.010001 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:43:32.009979    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86\": container with ID starting with 55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86 not found: ID does not exist" containerID="55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86"
Oct 30 03:43:32.010079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.010016    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86"} err="failed to get container status \"55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86\": rpc error: code = NotFound desc = could not find container \"55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86\": container with ID starting with 55cbd4bcd414839918a94162bccfd12dacc18a35d2052c50ccfc4938b9302f86 not found: ID does not exist"
Oct 30 03:43:32.010079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.010041    4377 scope.go:122] "RemoveContainer" containerID="34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b"
Oct 30 03:43:32.010299 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:43:32.010281    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b\": container with ID starting with 34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b not found: ID does not exist" containerID="34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b"
Oct 30 03:43:32.010339 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.010306    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b"} err="failed to get container status \"34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b\": rpc error: code = NotFound desc = could not find container \"34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b\": container with ID starting with 34cf130ffb23dfdcfd2b3cdd9f2ef7d2d17551bdfb9c05dda97562e431b1362b not found: ID does not exist"
Oct 30 03:43:32.010339 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.010324    4377 scope.go:122] "RemoveContainer" containerID="d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c"
Oct 30 03:43:32.010594 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:43:32.010577    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c\": container with ID starting with d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c not found: ID does not exist" containerID="d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c"
Oct 30 03:43:32.010657 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.010604    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c"} err="failed to get container status \"d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c\": rpc error: code = NotFound desc = could not find container \"d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c\": container with ID starting with d12087a07b7063800159ba77acb6d4925c302ca2e843cd1a1c0e9f9006d8466c not found: ID does not exist"
Oct 30 03:43:32.038044 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.038008    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"]
Oct 30 03:43:32.043938 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.043913    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-promote-v2-predictor-7d54ffb6dd-dtdh4"]
Oct 30 03:43:32.515206 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:32.515169    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" path="/var/lib/kubelet/pods/3487ecc1-6243-44ed-a13e-2c5ade7ddae7/volumes"
Oct 30 03:43:34.438346 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438310    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"]
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438664    4377 cpu_manager.go:414] "containerMap: removing container" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="storage-initializer"
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438677    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="storage-initializer"
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438687    4377 cpu_manager.go:414] "containerMap: removing container" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container"
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438692    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container"
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438698    4377 cpu_manager.go:414] "containerMap: removing container" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kube-rbac-proxy"
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438703    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kube-rbac-proxy"
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438766    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kube-rbac-proxy"
Oct 30 03:43:34.438789 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.438773    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="3487ecc1-6243-44ed-a13e-2c5ade7ddae7" containerName="kserve-container"
Oct 30 03:43:34.441693 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.441676    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.459764 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.459743    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Oct 30 03:43:34.472645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.472609    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Oct 30 03:43:34.472645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.472628    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:43:34.472645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.472610    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:43:34.474987 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.474975    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:43:34.475035 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.474992    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"]
Oct 30 03:43:34.480982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.480961    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:43:34.481173 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.481162    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:43:34.499079 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.499056    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c03efa9c-3310-46f6-91d3-14cf9269402f-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.499298 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.499102    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.499298 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.499122    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c03efa9c-3310-46f6-91d3-14cf9269402f-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.499298 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.499289    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhhh2\" (UniqueName: \"kubernetes.io/projected/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-api-access-nhhh2\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.499453 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.499338    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.589579 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.589545    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"]
Oct 30 03:43:34.592805 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.592786    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.600812 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.600788    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.600901 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.600830    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c03efa9c-3310-46f6-91d3-14cf9269402f-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.600901 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.600881    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.600994 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.600910    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c03efa9c-3310-46f6-91d3-14cf9269402f-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.600994 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.600966    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhhh2\" (UniqueName: \"kubernetes.io/projected/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-api-access-nhhh2\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.601358 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.601334    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c03efa9c-3310-46f6-91d3-14cf9269402f-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.601571 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.601547    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.601571 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.601558    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.603439 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.603421    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c03efa9c-3310-46f6-91d3-14cf9269402f-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.619575 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.616280    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"]
Oct 30 03:43:34.624900 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.624870    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhhh2\" (UniqueName: \"kubernetes.io/projected/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-api-access-nhhh2\") pod \"isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.660585 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.660561    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-rollback-predictor-serving-cert\""
Oct 30 03:43:34.702096 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.702024    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.702096 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.702063    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcsb5\" (UniqueName: \"kubernetes.io/projected/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-api-access-rcsb5\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.702234 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.702168    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c164ba2-703b-49e1-8ca7-f563b23b77da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.702234 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.702191    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.702234 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.702223    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c164ba2-703b-49e1-8ca7-f563b23b77da-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.758712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.758673    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:34.802892 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.802856    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.802892 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.802895    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcsb5\" (UniqueName: \"kubernetes.io/projected/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-api-access-rcsb5\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.803131 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.803041    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c164ba2-703b-49e1-8ca7-f563b23b77da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.803131 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.803108    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.803261 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.803168    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c164ba2-703b-49e1-8ca7-f563b23b77da-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.803388 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.803367    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c164ba2-703b-49e1-8ca7-f563b23b77da-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.803709 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.803688    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.803793 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.803708    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.814109 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.814082    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c164ba2-703b-49e1-8ca7-f563b23b77da-proxy-tls\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.828149 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.828123    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcsb5\" (UniqueName: \"kubernetes.io/projected/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-api-access-rcsb5\") pod \"isvc-canary-rollback-predictor-5769c8765-prbl9\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") " pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:34.909152 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:34.909116    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:35.072124 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:35.072094    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"]
Oct 30 03:43:35.074565 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:43:35.074535    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03efa9c_3310_46f6_91d3_14cf9269402f.slice/crio-025afab530c6b8ed6cac9301199a5fe27d0f1e41231fa65d04c5da0e2651e0d5 WatchSource:0}: Error finding container 025afab530c6b8ed6cac9301199a5fe27d0f1e41231fa65d04c5da0e2651e0d5: Status 404 returned error can't find the container with id 025afab530c6b8ed6cac9301199a5fe27d0f1e41231fa65d04c5da0e2651e0d5
Oct 30 03:43:35.210153 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:35.210070    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"]
Oct 30 03:43:35.213730 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:43:35.213696    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c164ba2_703b_49e1_8ca7_f563b23b77da.slice/crio-9275a9a4bab8f01e4dc052e2782af6ce3aceb1af91e384ed906825ec7efca0d2 WatchSource:0}: Error finding container 9275a9a4bab8f01e4dc052e2782af6ce3aceb1af91e384ed906825ec7efca0d2: Status 404 returned error can't find the container with id 9275a9a4bab8f01e4dc052e2782af6ce3aceb1af91e384ed906825ec7efca0d2
Oct 30 03:43:36.004417 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:36.004374    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerStarted","Data":"b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9"}
Oct 30 03:43:36.004909 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:36.004452    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerStarted","Data":"9275a9a4bab8f01e4dc052e2782af6ce3aceb1af91e384ed906825ec7efca0d2"}
Oct 30 03:43:36.005889 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:36.005863    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerStarted","Data":"523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3"}
Oct 30 03:43:36.005984 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:36.005912    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerStarted","Data":"025afab530c6b8ed6cac9301199a5fe27d0f1e41231fa65d04c5da0e2651e0d5"}
Oct 30 03:43:39.020664 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:39.020628    4377 generic.go:356] "Generic (PLEG): container finished" podID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerID="b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9" exitCode=0
Oct 30 03:43:39.021063 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:39.020709    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerDied","Data":"b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9"}
Oct 30 03:43:40.026934 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:40.026894    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerStarted","Data":"b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996"}
Oct 30 03:43:40.027377 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:40.026963    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerStarted","Data":"a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5"}
Oct 30 03:43:40.027377 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:40.027176    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:40.028532 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:40.028499    4377 generic.go:356] "Generic (PLEG): container finished" podID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerID="523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3" exitCode=0
Oct 30 03:43:40.028641 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:40.028570    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerDied","Data":"523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3"}
Oct 30 03:43:40.086338 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:40.086288    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podStartSLOduration=2.44192255 podStartE2EDuration="6.08627355s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.644351s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:40.084979864 +0000 UTC m=+2919.677308673" watchObservedRunningTime="2026-10-30 03:43:40.08627355 +0000 UTC m=+2919.678602356"
Oct 30 03:43:40.259320 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:43:40.259282    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:43:41.034593 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.034550    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerStarted","Data":"2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef"}
Oct 30 03:43:41.034958 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.034660    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerStarted","Data":"dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60"}
Oct 30 03:43:41.034958 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.034936    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:41.035310 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.035278    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:41.035439 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.035339    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:41.036237 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.036211    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:43:41.036332 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.036292    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:43:41.108811 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:41.108761    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podStartSLOduration=3.1673889060000002 podStartE2EDuration="7.108750384s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.941361478s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:43:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:43:41.108222218 +0000 UTC m=+2920.700551025" watchObservedRunningTime="2026-10-30 03:43:41.108750384 +0000 UTC m=+2920.701079190"
Oct 30 03:43:42.039140 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:42.039103    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:43:42.039565 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:42.039234    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:43:47.046038 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:47.045946    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:43:47.046537 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:47.046099    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:43:47.047566 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:47.047501    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:43:47.047710 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:47.047634    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:43:57.047581 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:57.047538    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:43:57.047581 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:43:57.047560    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:44:07.047424 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:07.047386    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:44:07.047943 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:07.047383    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:44:10.258897 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:44:10.258855    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:44:17.046937 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:17.046894    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:44:17.047356 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:17.046895    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:44:27.046736 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:27.046688    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:44:27.047120 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:27.046701    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:44:37.046687 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:37.046646    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:44:37.047160 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:37.046646    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:44:40.258813 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:44:40.258770    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:44:47.046952 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:47.046911    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:44:47.047359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:47.046912    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:44:55.509731 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:55.509690    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:44:57.049098 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:44:57.049065    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:45:05.511443 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:05.511411    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:45:10.259348 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:45:10.259311    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:45:14.358809 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:14.358730    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"]
Oct 30 03:45:14.361060 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:14.358962    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" containerID="cri-o://dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60" gracePeriod=30
Oct 30 03:45:14.361060 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:14.359010    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kube-rbac-proxy" containerID="cri-o://2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef" gracePeriod=30
Oct 30 03:45:15.435487 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:15.435453    4377 generic.go:356] "Generic (PLEG): container finished" podID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerID="2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef" exitCode=0
Oct 30 03:45:15.435928 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:15.435508    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerDied","Data":"2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef"}
Oct 30 03:45:15.510506 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:15.510464    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Oct 30 03:45:17.039904 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:17.039849    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Oct 30 03:45:19.404453 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.404428    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:45:19.454244 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.454168    4377 generic.go:356] "Generic (PLEG): container finished" podID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerID="dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60" exitCode=0
Oct 30 03:45:19.454244 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.454203    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerDied","Data":"dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60"}
Oct 30 03:45:19.454244 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.454225    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"
Oct 30 03:45:19.454453 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.454249    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl" event={"ID":"c03efa9c-3310-46f6-91d3-14cf9269402f","Type":"ContainerDied","Data":"025afab530c6b8ed6cac9301199a5fe27d0f1e41231fa65d04c5da0e2651e0d5"}
Oct 30 03:45:19.454453 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.454275    4377 scope.go:122] "RemoveContainer" containerID="2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef"
Oct 30 03:45:19.462368 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.462337    4377 scope.go:122] "RemoveContainer" containerID="dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60"
Oct 30 03:45:19.473083 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.473058    4377 scope.go:122] "RemoveContainer" containerID="523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3"
Oct 30 03:45:19.480840 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.480822    4377 scope.go:122] "RemoveContainer" containerID="2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef"
Oct 30 03:45:19.481136 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:45:19.481111    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef\": container with ID starting with 2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef not found: ID does not exist" containerID="2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef"
Oct 30 03:45:19.481195 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.481151    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef"} err="failed to get container status \"2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef\": rpc error: code = NotFound desc = could not find container \"2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef\": container with ID starting with 2d403f617ab8c0bc3c4d0577dd000ada6e7c757b051afd75292805a81b0aa0ef not found: ID does not exist"
Oct 30 03:45:19.481195 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.481175    4377 scope.go:122] "RemoveContainer" containerID="dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60"
Oct 30 03:45:19.481440 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:45:19.481418    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60\": container with ID starting with dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60 not found: ID does not exist" containerID="dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60"
Oct 30 03:45:19.481510 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.481449    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60"} err="failed to get container status \"dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60\": rpc error: code = NotFound desc = could not find container \"dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60\": container with ID starting with dd06f169f4c9d141f35195d48e1c0574fb11c618aadce3ec9dc52565aec1fd60 not found: ID does not exist"
Oct 30 03:45:19.481510 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.481471    4377 scope.go:122] "RemoveContainer" containerID="523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3"
Oct 30 03:45:19.481730 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:45:19.481711    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3\": container with ID starting with 523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3 not found: ID does not exist" containerID="523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3"
Oct 30 03:45:19.481783 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.481738    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3"} err="failed to get container status \"523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3\": rpc error: code = NotFound desc = could not find container \"523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3\": container with ID starting with 523b0c073f6f530975d6aa466c3e8a01539893f462a8f537912704bfe3838ae3 not found: ID does not exist"
Oct 30 03:45:19.582384 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582343    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-cabundle-cert\") pod \"c03efa9c-3310-46f6-91d3-14cf9269402f\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") "
Oct 30 03:45:19.582585 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582409    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-rbac-proxy-sar-config\") pod \"c03efa9c-3310-46f6-91d3-14cf9269402f\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") "
Oct 30 03:45:19.582585 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582439    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-api-access-nhhh2\" (UniqueName: \"kubernetes.io/projected/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-api-access-nhhh2\") pod \"c03efa9c-3310-46f6-91d3-14cf9269402f\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") "
Oct 30 03:45:19.582585 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582472    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c03efa9c-3310-46f6-91d3-14cf9269402f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c03efa9c-3310-46f6-91d3-14cf9269402f-kserve-provision-location\") pod \"c03efa9c-3310-46f6-91d3-14cf9269402f\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") "
Oct 30 03:45:19.582585 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582516    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c03efa9c-3310-46f6-91d3-14cf9269402f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c03efa9c-3310-46f6-91d3-14cf9269402f-proxy-tls\") pod \"c03efa9c-3310-46f6-91d3-14cf9269402f\" (UID: \"c03efa9c-3310-46f6-91d3-14cf9269402f\") "
Oct 30 03:45:19.582837 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582818    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c03efa9c-3310-46f6-91d3-14cf9269402f-kserve-provision-location" pod "c03efa9c-3310-46f6-91d3-14cf9269402f" (UID: "c03efa9c-3310-46f6-91d3-14cf9269402f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:45:19.582897 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582869    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-rbac-proxy-sar-config" pod "c03efa9c-3310-46f6-91d3-14cf9269402f" (UID: "c03efa9c-3310-46f6-91d3-14cf9269402f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:19.582957 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.582870    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-cabundle-cert" pod "c03efa9c-3310-46f6-91d3-14cf9269402f" (UID: "c03efa9c-3310-46f6-91d3-14cf9269402f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:45:19.584608 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.584581    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c03efa9c-3310-46f6-91d3-14cf9269402f-proxy-tls" pod "c03efa9c-3310-46f6-91d3-14cf9269402f" (UID: "c03efa9c-3310-46f6-91d3-14cf9269402f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:45:19.584701 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.584668    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-api-access-nhhh2" pod "c03efa9c-3310-46f6-91d3-14cf9269402f" (UID: "c03efa9c-3310-46f6-91d3-14cf9269402f"). InnerVolumeSpecName "kube-api-access-nhhh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:45:19.684029 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.683990    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:45:19.684029 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.684021    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:45:19.684029 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.684033    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhhh2\" (UniqueName: \"kubernetes.io/projected/c03efa9c-3310-46f6-91d3-14cf9269402f-kube-api-access-nhhh2\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:45:19.684262 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.684047    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c03efa9c-3310-46f6-91d3-14cf9269402f-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:45:19.684262 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.684060    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c03efa9c-3310-46f6-91d3-14cf9269402f-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:45:19.800466 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.800432    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"]
Oct 30 03:45:19.812866 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:19.812842    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-v2-predictor-54b7f68b8-lsztl"]
Oct 30 03:45:20.513962 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:45:20.513930    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" path="/var/lib/kubelet/pods/c03efa9c-3310-46f6-91d3-14cf9269402f/volumes"
Oct 30 03:45:40.259113 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:45:40.259072    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:46:10.258984 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:46:10.258945    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:46:40.258904 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:46:40.258866    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:46:44.771581 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:44.771485    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"]
Oct 30 03:46:44.772087 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:44.771759    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" containerID="cri-o://a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5" gracePeriod=30
Oct 30 03:46:44.772087 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:44.771798    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kube-rbac-proxy" containerID="cri-o://b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996" gracePeriod=30
Oct 30 03:46:45.791367 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:45.791333    4377 generic.go:356] "Generic (PLEG): container finished" podID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerID="b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996" exitCode=0
Oct 30 03:46:45.791790 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:45.791395    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerDied","Data":"b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996"}
Oct 30 03:46:47.040232 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:47.040192    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Oct 30 03:46:47.046672 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:47.046646    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Oct 30 03:46:49.517190 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.517164    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:46:49.609469 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.609387    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c164ba2-703b-49e1-8ca7-f563b23b77da-proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c164ba2-703b-49e1-8ca7-f563b23b77da-proxy-tls\") pod \"4c164ba2-703b-49e1-8ca7-f563b23b77da\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") "
Oct 30 03:46:49.609469 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.609434    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-cabundle-cert\") pod \"4c164ba2-703b-49e1-8ca7-f563b23b77da\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") "
Oct 30 03:46:49.609719 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.609494    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c164ba2-703b-49e1-8ca7-f563b23b77da-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c164ba2-703b-49e1-8ca7-f563b23b77da-kserve-provision-location\") pod \"4c164ba2-703b-49e1-8ca7-f563b23b77da\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") "
Oct 30 03:46:49.609719 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.609533    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-api-access-rcsb5\" (UniqueName: \"kubernetes.io/projected/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-api-access-rcsb5\") pod \"4c164ba2-703b-49e1-8ca7-f563b23b77da\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") "
Oct 30 03:46:49.609719 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.609582    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-rbac-proxy-sar-config\") pod \"4c164ba2-703b-49e1-8ca7-f563b23b77da\" (UID: \"4c164ba2-703b-49e1-8ca7-f563b23b77da\") "
Oct 30 03:46:49.610048 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.609835    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c164ba2-703b-49e1-8ca7-f563b23b77da-kserve-provision-location" pod "4c164ba2-703b-49e1-8ca7-f563b23b77da" (UID: "4c164ba2-703b-49e1-8ca7-f563b23b77da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:46:49.610048 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.609947    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-cabundle-cert" pod "4c164ba2-703b-49e1-8ca7-f563b23b77da" (UID: "4c164ba2-703b-49e1-8ca7-f563b23b77da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:49.610048 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.610029    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-rbac-proxy-sar-config" pod "4c164ba2-703b-49e1-8ca7-f563b23b77da" (UID: "4c164ba2-703b-49e1-8ca7-f563b23b77da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:46:49.611700 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.611680    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c164ba2-703b-49e1-8ca7-f563b23b77da-proxy-tls" pod "4c164ba2-703b-49e1-8ca7-f563b23b77da" (UID: "4c164ba2-703b-49e1-8ca7-f563b23b77da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:46:49.611802 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.611723    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-api-access-rcsb5" pod "4c164ba2-703b-49e1-8ca7-f563b23b77da" (UID: "4c164ba2-703b-49e1-8ca7-f563b23b77da"). InnerVolumeSpecName "kube-api-access-rcsb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:46:49.711067 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.711034    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:46:49.711067 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.711064    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4c164ba2-703b-49e1-8ca7-f563b23b77da-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:46:49.711067 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.711074    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4c164ba2-703b-49e1-8ca7-f563b23b77da-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:46:49.711287 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.711083    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4c164ba2-703b-49e1-8ca7-f563b23b77da-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:46:49.711287 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.711092    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcsb5\" (UniqueName: \"kubernetes.io/projected/4c164ba2-703b-49e1-8ca7-f563b23b77da-kube-api-access-rcsb5\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:46:49.811010 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.810978    4377 generic.go:356] "Generic (PLEG): container finished" podID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerID="a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5" exitCode=0
Oct 30 03:46:49.811167 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.811031    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerDied","Data":"a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5"}
Oct 30 03:46:49.811167 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.811042    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"
Oct 30 03:46:49.811167 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.811081    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9" event={"ID":"4c164ba2-703b-49e1-8ca7-f563b23b77da","Type":"ContainerDied","Data":"9275a9a4bab8f01e4dc052e2782af6ce3aceb1af91e384ed906825ec7efca0d2"}
Oct 30 03:46:49.811167 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.811150    4377 scope.go:122] "RemoveContainer" containerID="b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996"
Oct 30 03:46:49.819327 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.819313    4377 scope.go:122] "RemoveContainer" containerID="a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5"
Oct 30 03:46:49.826211 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.826195    4377 scope.go:122] "RemoveContainer" containerID="b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9"
Oct 30 03:46:49.833106 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.833089    4377 scope.go:122] "RemoveContainer" containerID="b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996"
Oct 30 03:46:49.833357 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:46:49.833332    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996\": container with ID starting with b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996 not found: ID does not exist" containerID="b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996"
Oct 30 03:46:49.833449 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.833363    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996"} err="failed to get container status \"b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996\": rpc error: code = NotFound desc = could not find container \"b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996\": container with ID starting with b418aec3acea214c66c4df0376db86797ccb031f0d91d5b02e77457d1851b996 not found: ID does not exist"
Oct 30 03:46:49.833449 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.833379    4377 scope.go:122] "RemoveContainer" containerID="a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5"
Oct 30 03:46:49.833685 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:46:49.833653    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5\": container with ID starting with a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5 not found: ID does not exist" containerID="a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5"
Oct 30 03:46:49.833728 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.833690    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5"} err="failed to get container status \"a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5\": rpc error: code = NotFound desc = could not find container \"a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5\": container with ID starting with a537eb6cebc9b90e2e930fb5b901bf8bded5e8cc6ee46c8a609dcbb24476e3e5 not found: ID does not exist"
Oct 30 03:46:49.833728 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.833704    4377 scope.go:122] "RemoveContainer" containerID="b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9"
Oct 30 03:46:49.833923 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:46:49.833907    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9\": container with ID starting with b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9 not found: ID does not exist" containerID="b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9"
Oct 30 03:46:49.833963 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.833927    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9"} err="failed to get container status \"b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9\": rpc error: code = NotFound desc = could not find container \"b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9\": container with ID starting with b36068939f190e95e764f20bb8807768206638f7832d99438dc4d565467ea6c9 not found: ID does not exist"
Oct 30 03:46:49.864000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.863928    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"]
Oct 30 03:46:49.876474 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:49.876447    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-rollback-predictor-5769c8765-prbl9"]
Oct 30 03:46:50.513645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:50.513607    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" path="/var/lib/kubelet/pods/4c164ba2-703b-49e1-8ca7-f563b23b77da/volumes"
Oct 30 03:46:52.473253 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473218    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"]
Oct 30 03:46:52.473653 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473623    4377 cpu_manager.go:414] "containerMap: removing container" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="storage-initializer"
Oct 30 03:46:52.473653 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473639    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="storage-initializer"
Oct 30 03:46:52.473653 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473649    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container"
Oct 30 03:46:52.473653 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473655    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473663    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="storage-initializer"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473668    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="storage-initializer"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473675    4377 cpu_manager.go:414] "containerMap: removing container" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473681    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473692    4377 cpu_manager.go:414] "containerMap: removing container" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kube-rbac-proxy"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473696    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kube-rbac-proxy"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473714    4377 cpu_manager.go:414] "containerMap: removing container" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kube-rbac-proxy"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473718    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kube-rbac-proxy"
Oct 30 03:46:52.473782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473777    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kube-rbac-proxy"
Oct 30 03:46:52.474033 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473787    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kube-rbac-proxy"
Oct 30 03:46:52.474033 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473794    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="c03efa9c-3310-46f6-91d3-14cf9269402f" containerName="kserve-container"
Oct 30 03:46:52.474033 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.473799    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c164ba2-703b-49e1-8ca7-f563b23b77da" containerName="kserve-container"
Oct 30 03:46:52.478377 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.478358    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.507957 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.507893    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:46:52.507957 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.507946    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:46:52.508159 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.508013    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Oct 30 03:46:52.510738 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.510721    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:46:52.510850 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.510751    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:46:52.510850 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.510774    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Oct 30 03:46:52.510850 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.510772    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:46:52.514744 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.514724    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"]
Oct 30 03:46:52.536225 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.536192    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.536354 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.536232    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca46fd5-46c4-40e9-8e04-f1711a30d016-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.536354 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.536333    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.536440 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.536364    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca46fd5-46c4-40e9-8e04-f1711a30d016-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.536440 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.536385    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhcbh\" (UniqueName: \"kubernetes.io/projected/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-api-access-fhcbh\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637167 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637128    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637174    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca46fd5-46c4-40e9-8e04-f1711a30d016-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637202    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhcbh\" (UniqueName: \"kubernetes.io/projected/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-api-access-fhcbh\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637290    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637321    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca46fd5-46c4-40e9-8e04-f1711a30d016-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637623 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637582    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca46fd5-46c4-40e9-8e04-f1711a30d016-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637909 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637889    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.637966 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.637943    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.639662 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.639641    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca46fd5-46c4-40e9-8e04-f1711a30d016-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.663202 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.663176    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhcbh\" (UniqueName: \"kubernetes.io/projected/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-api-access-fhcbh\") pod \"isvc-canary-stop-v2-predictor-854f888b88-sw2q9\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.771122 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.771033    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"]
Oct 30 03:46:52.774782 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.774761    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.793636 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.793616    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-canary-stop-predictor-serving-cert\""
Oct 30 03:46:52.795796 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.795778    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:52.799764 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.799740    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"]
Oct 30 03:46:52.838590 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.838557    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-cabundle-cert\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.838752 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.838597    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.838752 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.838704    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbe5849-97f8-48c6-adc8-35b2354da2f0-proxy-tls\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.838826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.838792    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdrfg\" (UniqueName: \"kubernetes.io/projected/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-api-access-cdrfg\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.838826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.838819    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kserve-provision-location\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.940226 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.940196    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdrfg\" (UniqueName: \"kubernetes.io/projected/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-api-access-cdrfg\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.940226 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.940230    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kserve-provision-location\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.940537 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.940255    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-cabundle-cert\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.940537 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.940286    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.940537 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.940317    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbe5849-97f8-48c6-adc8-35b2354da2f0-proxy-tls\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.940713 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.940690    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kserve-provision-location\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.941091 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.941069    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-cabundle-cert\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.941133 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.941069    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.942663 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.942642    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbe5849-97f8-48c6-adc8-35b2354da2f0-proxy-tls\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:52.963715 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:52.963694    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdrfg\" (UniqueName: \"kubernetes.io/projected/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-api-access-cdrfg\") pod \"isvc-canary-stop-predictor-574b8f9c57-v9wds\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") " pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:53.092255 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:53.092221    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:53.107356 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:53.107327    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"]
Oct 30 03:46:53.110701 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:46:53.110670    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeca46fd5_46c4_40e9_8e04_f1711a30d016.slice/crio-2e81bbbf37bf934557508456339d5cf0bc522a148422ec52b5e356c6b9894568 WatchSource:0}: Error finding container 2e81bbbf37bf934557508456339d5cf0bc522a148422ec52b5e356c6b9894568: Status 404 returned error can't find the container with id 2e81bbbf37bf934557508456339d5cf0bc522a148422ec52b5e356c6b9894568
Oct 30 03:46:53.411157 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:53.411127    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"]
Oct 30 03:46:53.413119 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:46:53.413090    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbe5849_97f8_48c6_adc8_35b2354da2f0.slice/crio-201cbc1fd1ded311d3f33c2c9cd0798d5cf9a87bcb92f3a7adbdce917545c148 WatchSource:0}: Error finding container 201cbc1fd1ded311d3f33c2c9cd0798d5cf9a87bcb92f3a7adbdce917545c148: Status 404 returned error can't find the container with id 201cbc1fd1ded311d3f33c2c9cd0798d5cf9a87bcb92f3a7adbdce917545c148
Oct 30 03:46:53.829118 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:53.829084    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerStarted","Data":"99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de"}
Oct 30 03:46:53.829632 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:53.829149    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerStarted","Data":"201cbc1fd1ded311d3f33c2c9cd0798d5cf9a87bcb92f3a7adbdce917545c148"}
Oct 30 03:46:53.830924 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:53.830898    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerStarted","Data":"698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a"}
Oct 30 03:46:53.831043 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:53.830947    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerStarted","Data":"2e81bbbf37bf934557508456339d5cf0bc522a148422ec52b5e356c6b9894568"}
Oct 30 03:46:57.849974 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:57.849941    4377 generic.go:356] "Generic (PLEG): container finished" podID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerID="99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de" exitCode=0
Oct 30 03:46:57.850416 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:57.850009    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerDied","Data":"99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de"}
Oct 30 03:46:57.851721 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:57.851693    4377 generic.go:356] "Generic (PLEG): container finished" podID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerID="698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a" exitCode=0
Oct 30 03:46:57.851859 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:57.851720    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerDied","Data":"698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a"}
Oct 30 03:46:58.858309 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.858279    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerStarted","Data":"58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c"}
Oct 30 03:46:58.858790 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.858342    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerStarted","Data":"cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003"}
Oct 30 03:46:58.858790 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.858650    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:58.858923 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.858802    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:46:58.860106 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.860079    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:46:58.860708 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.860688    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerStarted","Data":"761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70"}
Oct 30 03:46:58.860781 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.860734    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerStarted","Data":"0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add"}
Oct 30 03:46:58.860925 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.860908    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:58.904047 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.904002    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podStartSLOduration=3.058260793 podStartE2EDuration="6.903989409s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845728616s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:58.90297228 +0000 UTC m=+3118.495301085" watchObservedRunningTime="2026-10-30 03:46:58.903989409 +0000 UTC m=+3118.496318216"
Oct 30 03:46:58.949172 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:58.949129    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podStartSLOduration=3.118771326 podStartE2EDuration="6.949115288s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.830343962s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:46:58.948044655 +0000 UTC m=+3118.540373461" watchObservedRunningTime="2026-10-30 03:46:58.949115288 +0000 UTC m=+3118.541444093"
Oct 30 03:46:59.865372 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:59.865330    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:46:59.865372 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:59.865346    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:46:59.866567 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:46:59.866547    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:47:00.870012 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:00.869973    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:47:04.870118 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:04.870089    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:47:04.870576 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:04.870550    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:47:05.875040 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:05.875013    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:47:05.875538 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:05.875494    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:47:10.258593 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:47:10.258553    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:47:14.870722 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:14.870674    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:47:15.876090 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:15.876045    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:47:24.871348 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:24.871304    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:47:25.876029 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:25.875991    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:47:34.870565 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:34.870504    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:47:35.875890 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:35.875853    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:47:40.258535 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:47:40.258484    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:47:44.871138 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:44.871091    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:47:45.876072 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:45.876032    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:47:54.871175 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:54.871135    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:47:55.875451 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:47:55.875413    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:48:04.870567 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:04.870499    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:48:05.875916 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:05.875872    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:48:10.258873 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:10.258822    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:48:14.871457 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:14.871365    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Oct 30 03:48:15.875636 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:15.875599    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:48:24.871730 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:24.871696    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:48:25.876685 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:25.876656    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:48:26.315506 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:26.315474    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"]
Oct 30 03:48:26.315751 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:26.315725    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container" containerID="cri-o://cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003" gracePeriod=30
Oct 30 03:48:26.315814 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:26.315753    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kube-rbac-proxy" containerID="cri-o://58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c" gracePeriod=30
Oct 30 03:48:27.222346 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:27.222304    4377 generic.go:356] "Generic (PLEG): container finished" podID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerID="58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c" exitCode=0
Oct 30 03:48:27.222791 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:27.222381    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerDied","Data":"58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c"}
Oct 30 03:48:29.866168 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:29.866111    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Oct 30 03:48:31.159051 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.159026    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:48:31.186932 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.186896    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-rbac-proxy-sar-config\") pod \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") "
Oct 30 03:48:31.187085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.186949    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbe5849-97f8-48c6-adc8-35b2354da2f0-proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbe5849-97f8-48c6-adc8-35b2354da2f0-proxy-tls\") pod \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") "
Oct 30 03:48:31.187085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.187013    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-cabundle-cert\") pod \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") "
Oct 30 03:48:31.187085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.187046    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-api-access-cdrfg\" (UniqueName: \"kubernetes.io/projected/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-api-access-cdrfg\") pod \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") "
Oct 30 03:48:31.187085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.187074    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kserve-provision-location\") pod \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\" (UID: \"3fbe5849-97f8-48c6-adc8-35b2354da2f0\") "
Oct 30 03:48:31.187338 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.187312    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-rbac-proxy-sar-config" pod "3fbe5849-97f8-48c6-adc8-35b2354da2f0" (UID: "3fbe5849-97f8-48c6-adc8-35b2354da2f0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:31.187460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.187432    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-cabundle-cert" pod "3fbe5849-97f8-48c6-adc8-35b2354da2f0" (UID: "3fbe5849-97f8-48c6-adc8-35b2354da2f0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:31.187546 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.187486    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kserve-provision-location" pod "3fbe5849-97f8-48c6-adc8-35b2354da2f0" (UID: "3fbe5849-97f8-48c6-adc8-35b2354da2f0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:31.189268 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.189248    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbe5849-97f8-48c6-adc8-35b2354da2f0-proxy-tls" pod "3fbe5849-97f8-48c6-adc8-35b2354da2f0" (UID: "3fbe5849-97f8-48c6-adc8-35b2354da2f0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:31.189351 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.189299    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-api-access-cdrfg" pod "3fbe5849-97f8-48c6-adc8-35b2354da2f0" (UID: "3fbe5849-97f8-48c6-adc8-35b2354da2f0"). InnerVolumeSpecName "kube-api-access-cdrfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:31.239712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.239680    4377 generic.go:356] "Generic (PLEG): container finished" podID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerID="cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003" exitCode=0
Oct 30 03:48:31.239866 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.239742    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"
Oct 30 03:48:31.239866 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.239747    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerDied","Data":"cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003"}
Oct 30 03:48:31.239866 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.239811    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds" event={"ID":"3fbe5849-97f8-48c6-adc8-35b2354da2f0","Type":"ContainerDied","Data":"201cbc1fd1ded311d3f33c2c9cd0798d5cf9a87bcb92f3a7adbdce917545c148"}
Oct 30 03:48:31.239866 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.239841    4377 scope.go:122] "RemoveContainer" containerID="58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c"
Oct 30 03:48:31.249397 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.249361    4377 scope.go:122] "RemoveContainer" containerID="cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003"
Oct 30 03:48:31.258725 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.258707    4377 scope.go:122] "RemoveContainer" containerID="99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de"
Oct 30 03:48:31.266727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.266702    4377 scope.go:122] "RemoveContainer" containerID="58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c"
Oct 30 03:48:31.267047 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:31.267018    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c\": container with ID starting with 58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c not found: ID does not exist" containerID="58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c"
Oct 30 03:48:31.267132 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.267061    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c"} err="failed to get container status \"58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c\": rpc error: code = NotFound desc = could not find container \"58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c\": container with ID starting with 58697a1b4ba2e93a75e20ea1385b249c52f8ea944550457087dadeee523ac41c not found: ID does not exist"
Oct 30 03:48:31.267132 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.267087    4377 scope.go:122] "RemoveContainer" containerID="cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003"
Oct 30 03:48:31.267370 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:31.267348    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003\": container with ID starting with cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003 not found: ID does not exist" containerID="cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003"
Oct 30 03:48:31.267424 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.267375    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003"} err="failed to get container status \"cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003\": rpc error: code = NotFound desc = could not find container \"cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003\": container with ID starting with cbc63bcd8bbc023dca7177692e8d8150e63109fd5d59bb2b4acddb3bcf051003 not found: ID does not exist"
Oct 30 03:48:31.267424 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.267395    4377 scope.go:122] "RemoveContainer" containerID="99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de"
Oct 30 03:48:31.267660 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:31.267641    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de\": container with ID starting with 99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de not found: ID does not exist" containerID="99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de"
Oct 30 03:48:31.267723 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.267670    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de"} err="failed to get container status \"99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de\": rpc error: code = NotFound desc = could not find container \"99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de\": container with ID starting with 99a75e169862dd1e1f703dd6ba97b1d63ac009e3a91bbaec5993898d247e84de not found: ID does not exist"
Oct 30 03:48:31.287682 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.287658    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdrfg\" (UniqueName: \"kubernetes.io/projected/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-api-access-cdrfg\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:31.287797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.287685    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:31.287797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.287700    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:31.287797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.287714    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3fbe5849-97f8-48c6-adc8-35b2354da2f0-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:31.287797 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.287727    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3fbe5849-97f8-48c6-adc8-35b2354da2f0-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:31.320690 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.320664    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"]
Oct 30 03:48:31.336362 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:31.336342    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-predictor-574b8f9c57-v9wds"]
Oct 30 03:48:32.492001 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:32.491971    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"]
Oct 30 03:48:32.492405 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:32.492228    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" containerID="cri-o://0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add" gracePeriod=30
Oct 30 03:48:32.492405 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:32.492310    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kube-rbac-proxy" containerID="cri-o://761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70" gracePeriod=30
Oct 30 03:48:32.513767 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:32.513744    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" path="/var/lib/kubelet/pods/3fbe5849-97f8-48c6-adc8-35b2354da2f0/volumes"
Oct 30 03:48:33.251259 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:33.251220    4377 generic.go:356] "Generic (PLEG): container finished" podID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerID="761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70" exitCode=0
Oct 30 03:48:33.251259 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:33.251262    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerDied","Data":"761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70"}
Oct 30 03:48:35.871102 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:35.871064    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Oct 30 03:48:35.875416 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:35.875391    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Oct 30 03:48:37.432595 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.432571    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:48:37.540898 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.540867    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eca46fd5-46c4-40e9-8e04-f1711a30d016-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca46fd5-46c4-40e9-8e04-f1711a30d016-kserve-provision-location\") pod \"eca46fd5-46c4-40e9-8e04-f1711a30d016\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") "
Oct 30 03:48:37.541085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.540905    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-cabundle-cert\") pod \"eca46fd5-46c4-40e9-8e04-f1711a30d016\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") "
Oct 30 03:48:37.541085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.540967    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-rbac-proxy-sar-config\") pod \"eca46fd5-46c4-40e9-8e04-f1711a30d016\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") "
Oct 30 03:48:37.541085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.540989    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eca46fd5-46c4-40e9-8e04-f1711a30d016-proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca46fd5-46c4-40e9-8e04-f1711a30d016-proxy-tls\") pod \"eca46fd5-46c4-40e9-8e04-f1711a30d016\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") "
Oct 30 03:48:37.541085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.541026    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-api-access-fhcbh\" (UniqueName: \"kubernetes.io/projected/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-api-access-fhcbh\") pod \"eca46fd5-46c4-40e9-8e04-f1711a30d016\" (UID: \"eca46fd5-46c4-40e9-8e04-f1711a30d016\") "
Oct 30 03:48:37.541359 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.541265    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eca46fd5-46c4-40e9-8e04-f1711a30d016-kserve-provision-location" pod "eca46fd5-46c4-40e9-8e04-f1711a30d016" (UID: "eca46fd5-46c4-40e9-8e04-f1711a30d016"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:48:37.541479 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.541393    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-cabundle-cert" pod "eca46fd5-46c4-40e9-8e04-f1711a30d016" (UID: "eca46fd5-46c4-40e9-8e04-f1711a30d016"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:37.541479 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.541423    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-rbac-proxy-sar-config" pod "eca46fd5-46c4-40e9-8e04-f1711a30d016" (UID: "eca46fd5-46c4-40e9-8e04-f1711a30d016"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:48:37.543117 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.543097    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eca46fd5-46c4-40e9-8e04-f1711a30d016-proxy-tls" pod "eca46fd5-46c4-40e9-8e04-f1711a30d016" (UID: "eca46fd5-46c4-40e9-8e04-f1711a30d016"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:48:37.543207 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.543157    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-api-access-fhcbh" pod "eca46fd5-46c4-40e9-8e04-f1711a30d016" (UID: "eca46fd5-46c4-40e9-8e04-f1711a30d016"). InnerVolumeSpecName "kube-api-access-fhcbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:48:37.641920 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.641861    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eca46fd5-46c4-40e9-8e04-f1711a30d016-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:37.641920 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.641913    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:37.641920 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.641928    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:37.641920 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.641940    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eca46fd5-46c4-40e9-8e04-f1711a30d016-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:37.642230 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:37.641953    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhcbh\" (UniqueName: \"kubernetes.io/projected/eca46fd5-46c4-40e9-8e04-f1711a30d016-kube-api-access-fhcbh\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:48:38.272981 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.272945    4377 generic.go:356] "Generic (PLEG): container finished" podID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerID="0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add" exitCode=0
Oct 30 03:48:38.273190 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.273001    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"
Oct 30 03:48:38.273190 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.273024    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerDied","Data":"0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add"}
Oct 30 03:48:38.273190 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.273103    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9" event={"ID":"eca46fd5-46c4-40e9-8e04-f1711a30d016","Type":"ContainerDied","Data":"2e81bbbf37bf934557508456339d5cf0bc522a148422ec52b5e356c6b9894568"}
Oct 30 03:48:38.273190 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.273143    4377 scope.go:122] "RemoveContainer" containerID="761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70"
Oct 30 03:48:38.281756 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.281739    4377 scope.go:122] "RemoveContainer" containerID="0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add"
Oct 30 03:48:38.289038 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.289021    4377 scope.go:122] "RemoveContainer" containerID="698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a"
Oct 30 03:48:38.295991 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.295974    4377 scope.go:122] "RemoveContainer" containerID="761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70"
Oct 30 03:48:38.296244 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:38.296220    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70\": container with ID starting with 761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70 not found: ID does not exist" containerID="761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70"
Oct 30 03:48:38.296320 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.296259    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70"} err="failed to get container status \"761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70\": rpc error: code = NotFound desc = could not find container \"761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70\": container with ID starting with 761f01e6524dce4a86551054248d167b7c4c0cc6920f77e8b0f06a50bda50c70 not found: ID does not exist"
Oct 30 03:48:38.296320 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.296283    4377 scope.go:122] "RemoveContainer" containerID="0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add"
Oct 30 03:48:38.296569 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:38.296551    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add\": container with ID starting with 0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add not found: ID does not exist" containerID="0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add"
Oct 30 03:48:38.296613 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.296575    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add"} err="failed to get container status \"0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add\": rpc error: code = NotFound desc = could not find container \"0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add\": container with ID starting with 0fe2d18471063bd89ace988a4fe37a78c652c513389041f6882eb66a060b1add not found: ID does not exist"
Oct 30 03:48:38.296613 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.296590    4377 scope.go:122] "RemoveContainer" containerID="698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a"
Oct 30 03:48:38.296796 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:38.296782    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a\": container with ID starting with 698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a not found: ID does not exist" containerID="698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a"
Oct 30 03:48:38.296837 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.296799    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a"} err="failed to get container status \"698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a\": rpc error: code = NotFound desc = could not find container \"698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a\": container with ID starting with 698d03fc6165e6759c5ef3e10e9a5520b2a53e84ad4685b42cf66b0e0e99b52a not found: ID does not exist"
Oct 30 03:48:38.338568 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.338542    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"]
Oct 30 03:48:38.351179 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.351149    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-canary-stop-v2-predictor-854f888b88-sw2q9"]
Oct 30 03:48:38.513857 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:38.513827    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" path="/var/lib/kubelet/pods/eca46fd5-46c4-40e9-8e04-f1711a30d016/volumes"
Oct 30 03:48:39.347513 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347478    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"]
Oct 30 03:48:39.347829 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347817    4377 cpu_manager.go:414] "containerMap: removing container" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="storage-initializer"
Oct 30 03:48:39.347875 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347830    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="storage-initializer"
Oct 30 03:48:39.347875 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347841    4377 cpu_manager.go:414] "containerMap: removing container" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container"
Oct 30 03:48:39.347875 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347847    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container"
Oct 30 03:48:39.347875 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347863    4377 cpu_manager.go:414] "containerMap: removing container" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="storage-initializer"
Oct 30 03:48:39.347875 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347869    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="storage-initializer"
Oct 30 03:48:39.347875 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347876    4377 cpu_manager.go:414] "containerMap: removing container" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kube-rbac-proxy"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347880    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kube-rbac-proxy"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347890    4377 cpu_manager.go:414] "containerMap: removing container" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347895    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347904    4377 cpu_manager.go:414] "containerMap: removing container" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kube-rbac-proxy"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347909    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kube-rbac-proxy"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347953    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kube-rbac-proxy"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347962    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="eca46fd5-46c4-40e9-8e04-f1711a30d016" containerName="kserve-container"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347970    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kube-rbac-proxy"
Oct 30 03:48:39.348056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.347976    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbe5849-97f8-48c6-adc8-35b2354da2f0" containerName="kserve-container"
Oct 30 03:48:39.352365 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.352349    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.368466 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.368442    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:48:39.368466 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.368463    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:48:39.380804 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.380756    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Oct 30 03:48:39.380804 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.380799    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:48:39.381056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.380821    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:48:39.381056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.380926    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:48:39.381056 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.380956    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-predictor-serving-cert\""
Oct 30 03:48:39.382470 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.382449    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"]
Oct 30 03:48:39.459274 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.459241    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389387eb-6c19-4add-badb-11de8701cd5f-proxy-tls\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.459431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.459308    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdckx\" (UniqueName: \"kubernetes.io/projected/389387eb-6c19-4add-badb-11de8701cd5f-kube-api-access-bdckx\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.459431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.459329    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/389387eb-6c19-4add-badb-11de8701cd5f-kserve-provision-location\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.459431 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.459378    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-cabundle-cert\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.459563 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.459436    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.560365 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.560329    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdckx\" (UniqueName: \"kubernetes.io/projected/389387eb-6c19-4add-badb-11de8701cd5f-kube-api-access-bdckx\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.560365 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.560365    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/389387eb-6c19-4add-badb-11de8701cd5f-kserve-provision-location\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.560907 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.560531    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-cabundle-cert\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.560907 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.560608    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.560907 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.560656    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389387eb-6c19-4add-badb-11de8701cd5f-proxy-tls\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.560907 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.560781    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/389387eb-6c19-4add-badb-11de8701cd5f-kserve-provision-location\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.561325 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.561305    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-cabundle-cert\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.561365 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.561316    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.563070 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.563049    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389387eb-6c19-4add-badb-11de8701cd5f-proxy-tls\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.584270 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.584245    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdckx\" (UniqueName: \"kubernetes.io/projected/389387eb-6c19-4add-badb-11de8701cd5f-kube-api-access-bdckx\") pod \"isvc-lightgbm-predictor-554fcc9df-fzwgs\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.669052 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.668975    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:39.991692 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:39.991667    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"]
Oct 30 03:48:39.993696 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:48:39.993663    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod389387eb_6c19_4add_badb_11de8701cd5f.slice/crio-ee1965f21411ece423d00be1de31e120afdab2abdb9bb49ad954a2325a7d575a WatchSource:0}: Error finding container ee1965f21411ece423d00be1de31e120afdab2abdb9bb49ad954a2325a7d575a: Status 404 returned error can't find the container with id ee1965f21411ece423d00be1de31e120afdab2abdb9bb49ad954a2325a7d575a
Oct 30 03:48:40.259202 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:48:40.259103    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:48:40.283416 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:40.283372    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerStarted","Data":"7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c"}
Oct 30 03:48:40.283604 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:40.283454    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerStarted","Data":"ee1965f21411ece423d00be1de31e120afdab2abdb9bb49ad954a2325a7d575a"}
Oct 30 03:48:44.300727 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:44.300692    4377 generic.go:356] "Generic (PLEG): container finished" podID="389387eb-6c19-4add-badb-11de8701cd5f" containerID="7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c" exitCode=0
Oct 30 03:48:44.301105 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:44.300764    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerDied","Data":"7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c"}
Oct 30 03:48:44.301985 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:44.301969    4377 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:48:51.338343 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:51.338299    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerStarted","Data":"145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a"}
Oct 30 03:48:51.338766 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:51.338381    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerStarted","Data":"8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc"}
Oct 30 03:48:51.338766 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:51.338574    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:51.403009 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:51.402964    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podStartSLOduration=2.07660301 podStartE2EDuration="12.402951476s" totalImagesPullingTime="6.534700283s" totalInitContainerRuntime="3.791648183s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:48:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:48:51.401761386 +0000 UTC m=+3230.994090189" watchObservedRunningTime="2026-10-30 03:48:51.402951476 +0000 UTC m=+3230.995280282"
Oct 30 03:48:52.347575 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:52.347537    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:52.349052 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:52.349018    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:48:53.351024 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:53.350984    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:48:58.355566 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:58.355534    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:48:58.356176 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:48:58.356149    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:49:08.356363 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:49:08.356325    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:49:10.259453 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:49:10.259418    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:49:18.356776 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:49:18.356732    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:49:28.356110 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:49:28.356064    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:49:38.356536 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:49:38.356476    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:49:40.258118 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:49:40.258077    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:49:48.356133 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:49:48.356048    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:49:58.356262 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:49:58.356220    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:50:00.511619 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:00.511576    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:50:10.261490 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:50:10.261445    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:50:10.511772 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:10.511667    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:50:20.511630 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:20.511589    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:50:30.514077 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:30.514046    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:50:40.206850 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:40.206815    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"]
Oct 30 03:50:40.207314 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:40.207037    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" containerID="cri-o://8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc" gracePeriod=30
Oct 30 03:50:40.207314 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:40.207083    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kube-rbac-proxy" containerID="cri-o://145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a" gracePeriod=30
Oct 30 03:50:40.260168 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:50:40.260128    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:50:40.511824 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:40.511737    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Oct 30 03:50:40.777193 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:40.777163    4377 generic.go:356] "Generic (PLEG): container finished" podID="389387eb-6c19-4add-badb-11de8701cd5f" containerID="145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a" exitCode=0
Oct 30 03:50:40.777352 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:40.777218    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerDied","Data":"145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a"}
Oct 30 03:50:43.351712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:43.351672    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Oct 30 03:50:45.740623 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.740597    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:50:45.799098 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.799059    4377 generic.go:356] "Generic (PLEG): container finished" podID="389387eb-6c19-4add-badb-11de8701cd5f" containerID="8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc" exitCode=0
Oct 30 03:50:45.799249 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.799124    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerDied","Data":"8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc"}
Oct 30 03:50:45.799249 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.799138    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"
Oct 30 03:50:45.799249 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.799182    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs" event={"ID":"389387eb-6c19-4add-badb-11de8701cd5f","Type":"ContainerDied","Data":"ee1965f21411ece423d00be1de31e120afdab2abdb9bb49ad954a2325a7d575a"}
Oct 30 03:50:45.799249 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.799240    4377 scope.go:122] "RemoveContainer" containerID="145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a"
Oct 30 03:50:45.807389 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.807357    4377 scope.go:122] "RemoveContainer" containerID="8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc"
Oct 30 03:50:45.814892 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.814868    4377 scope.go:122] "RemoveContainer" containerID="7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c"
Oct 30 03:50:45.823323 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.823307    4377 scope.go:122] "RemoveContainer" containerID="145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a"
Oct 30 03:50:45.823561 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:50:45.823538    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a\": container with ID starting with 145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a not found: ID does not exist" containerID="145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a"
Oct 30 03:50:45.823625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.823572    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a"} err="failed to get container status \"145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a\": rpc error: code = NotFound desc = could not find container \"145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a\": container with ID starting with 145055ea811ede93de1373adcba4ba3e6af8ae5e46426378099aae3f5191896a not found: ID does not exist"
Oct 30 03:50:45.823625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.823589    4377 scope.go:122] "RemoveContainer" containerID="8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc"
Oct 30 03:50:45.823850 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:50:45.823831    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc\": container with ID starting with 8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc not found: ID does not exist" containerID="8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc"
Oct 30 03:50:45.823910 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.823859    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc"} err="failed to get container status \"8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc\": rpc error: code = NotFound desc = could not find container \"8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc\": container with ID starting with 8fd66a5e96bb09142e14ac317f84d6e46ad2279e9cf1ffc6e296af59b92462bc not found: ID does not exist"
Oct 30 03:50:45.823910 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.823880    4377 scope.go:122] "RemoveContainer" containerID="7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c"
Oct 30 03:50:45.824134 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:50:45.824120    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c\": container with ID starting with 7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c not found: ID does not exist" containerID="7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c"
Oct 30 03:50:45.824177 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.824136    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c"} err="failed to get container status \"7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c\": rpc error: code = NotFound desc = could not find container \"7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c\": container with ID starting with 7bdce79d7c3d24dbaa9d4458098193b5b0993522d28e32ae48cdcbd7141aad6c not found: ID does not exist"
Oct 30 03:50:45.831351 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831331    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389387eb-6c19-4add-badb-11de8701cd5f-kube-api-access-bdckx\" (UniqueName: \"kubernetes.io/projected/389387eb-6c19-4add-badb-11de8701cd5f-kube-api-access-bdckx\") pod \"389387eb-6c19-4add-badb-11de8701cd5f\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") "
Oct 30 03:50:45.831429 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831368    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389387eb-6c19-4add-badb-11de8701cd5f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/389387eb-6c19-4add-badb-11de8701cd5f-kserve-provision-location\") pod \"389387eb-6c19-4add-badb-11de8701cd5f\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") "
Oct 30 03:50:45.831476 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831461    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-kube-rbac-proxy-sar-config\") pod \"389387eb-6c19-4add-badb-11de8701cd5f\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") "
Oct 30 03:50:45.831512 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831500    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/389387eb-6c19-4add-badb-11de8701cd5f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/389387eb-6c19-4add-badb-11de8701cd5f-proxy-tls\") pod \"389387eb-6c19-4add-badb-11de8701cd5f\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") "
Oct 30 03:50:45.831570 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831546    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-cabundle-cert\") pod \"389387eb-6c19-4add-badb-11de8701cd5f\" (UID: \"389387eb-6c19-4add-badb-11de8701cd5f\") "
Oct 30 03:50:45.831757 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831732    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389387eb-6c19-4add-badb-11de8701cd5f-kserve-provision-location" pod "389387eb-6c19-4add-badb-11de8701cd5f" (UID: "389387eb-6c19-4add-badb-11de8701cd5f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:50:45.831881 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831854    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-kube-rbac-proxy-sar-config" pod "389387eb-6c19-4add-badb-11de8701cd5f" (UID: "389387eb-6c19-4add-badb-11de8701cd5f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:45.831948 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.831886    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-cabundle-cert" pod "389387eb-6c19-4add-badb-11de8701cd5f" (UID: "389387eb-6c19-4add-badb-11de8701cd5f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:50:45.833328 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.833307    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389387eb-6c19-4add-badb-11de8701cd5f-kube-api-access-bdckx" pod "389387eb-6c19-4add-badb-11de8701cd5f" (UID: "389387eb-6c19-4add-badb-11de8701cd5f"). InnerVolumeSpecName "kube-api-access-bdckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:50:45.833425 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.833388    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/389387eb-6c19-4add-badb-11de8701cd5f-proxy-tls" pod "389387eb-6c19-4add-badb-11de8701cd5f" (UID: "389387eb-6c19-4add-badb-11de8701cd5f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:50:45.932896 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.932831    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdckx\" (UniqueName: \"kubernetes.io/projected/389387eb-6c19-4add-badb-11de8701cd5f-kube-api-access-bdckx\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:50:45.932896 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.932854    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/389387eb-6c19-4add-badb-11de8701cd5f-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:50:45.932896 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.932865    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:50:45.932896 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.932873    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/389387eb-6c19-4add-badb-11de8701cd5f-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:50:45.932896 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:45.932883    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/389387eb-6c19-4add-badb-11de8701cd5f-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:50:46.144960 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:46.144928    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"]
Oct 30 03:50:46.157502 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:46.157473    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-predictor-554fcc9df-fzwgs"]
Oct 30 03:50:46.513599 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:46.513572    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389387eb-6c19-4add-badb-11de8701cd5f" path="/var/lib/kubelet/pods/389387eb-6c19-4add-badb-11de8701cd5f/volumes"
Oct 30 03:50:47.827578 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.827541    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"]
Oct 30 03:50:47.828086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.827971    4377 cpu_manager.go:414] "containerMap: removing container" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container"
Oct 30 03:50:47.828086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.827990    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container"
Oct 30 03:50:47.828086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.828029    4377 cpu_manager.go:414] "containerMap: removing container" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="storage-initializer"
Oct 30 03:50:47.828086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.828037    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="storage-initializer"
Oct 30 03:50:47.828086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.828051    4377 cpu_manager.go:414] "containerMap: removing container" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kube-rbac-proxy"
Oct 30 03:50:47.828086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.828059    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kube-rbac-proxy"
Oct 30 03:50:47.828404 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.828130    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kube-rbac-proxy"
Oct 30 03:50:47.828404 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.828144    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="389387eb-6c19-4add-badb-11de8701cd5f" containerName="kserve-container"
Oct 30 03:50:47.832612 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.832592    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:47.847982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.847957    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:50:47.848078 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.848045    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"kube-root-ca.crt\""
Oct 30 03:50:47.860478 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.860453    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"seaweedfs-s3-creds\""
Oct 30 03:50:47.860478 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.860460    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"openshift-service-ca.crt\""
Oct 30 03:50:47.860663 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.860490    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Oct 30 03:50:47.860663 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.860642    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"default-dockercfg-887kz\""
Oct 30 03:50:47.860737 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.860728    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-4a54dc1685\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Oct 30 03:50:47.862146 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.862130    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"]
Oct 30 03:50:47.948617 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.948586    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f25fw\" (UniqueName: \"kubernetes.io/projected/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-api-access-f25fw\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:47.948787 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.948637    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e73b3b1-1b43-4d29-a07a-1df947b6e116-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:47.948787 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.948712    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:47.948787 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.948765    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:47.948890 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:47.948790    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.049565 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.049510    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f25fw\" (UniqueName: \"kubernetes.io/projected/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-api-access-f25fw\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.049754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.049576    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e73b3b1-1b43-4d29-a07a-1df947b6e116-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.049754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.049598    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.049754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.049626    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.049754 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.049646    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.050085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.050036    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.050396 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.050357    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.050515 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.050368    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.052244 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.052224    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e73b3b1-1b43-4d29-a07a-1df947b6e116-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.073223 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.073192    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f25fw\" (UniqueName: \"kubernetes.io/projected/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-api-access-f25fw\") pod \"isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") " pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.150193 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.150112    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:48.463460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.463376    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"]
Oct 30 03:50:48.466256 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:50:48.466226    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e73b3b1_1b43_4d29_a07a_1df947b6e116.slice/crio-ef0fa6497f8da434c19bfdf0d2efac73b8d87cee1cfd66f337ac88646892fe58 WatchSource:0}: Error finding container ef0fa6497f8da434c19bfdf0d2efac73b8d87cee1cfd66f337ac88646892fe58: Status 404 returned error can't find the container with id ef0fa6497f8da434c19bfdf0d2efac73b8d87cee1cfd66f337ac88646892fe58
Oct 30 03:50:48.814218 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.814183    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerStarted","Data":"94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81"}
Oct 30 03:50:48.814425 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:48.814244    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerStarted","Data":"ef0fa6497f8da434c19bfdf0d2efac73b8d87cee1cfd66f337ac88646892fe58"}
Oct 30 03:50:52.833285 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:52.833245    4377 generic.go:356] "Generic (PLEG): container finished" podID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerID="94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81" exitCode=0
Oct 30 03:50:52.833705 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:52.833317    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerDied","Data":"94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81"}
Oct 30 03:50:53.839879 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:53.839844    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerStarted","Data":"a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536"}
Oct 30 03:50:53.840334 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:53.839903    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerStarted","Data":"5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a"}
Oct 30 03:50:53.840334 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:53.840226    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:53.840455 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:53.840351    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:53.841857 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:53.841831    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:50:53.905714 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:53.905668    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podStartSLOduration=3.272967917 podStartE2EDuration="6.905655693s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.632687776s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:50:53.904557375 +0000 UTC m=+3353.496886178" watchObservedRunningTime="2026-10-30 03:50:53.905655693 +0000 UTC m=+3353.497984499"
Oct 30 03:50:54.844333 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:54.844293    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:50:59.849661 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:59.849627    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:50:59.850227 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:50:59.850198    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:51:09.850156 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:51:09.850119    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:51:10.259147 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:51:10.259064    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:51:19.850608 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:51:19.850502    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:51:29.851060 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:51:29.851019    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:51:39.850700 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:51:39.850656    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:51:40.258475 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:51:40.258375    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:51:49.850556 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:51:49.850499    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:51:59.850272 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:51:59.850229    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:52:09.851137 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:09.851097    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:52:10.258922 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:52:10.258841    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:52:13.509468 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:13.509420    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:52:23.509683 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:23.509640    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:52:33.510257 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:33.510230    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:52:39.215955 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:39.215924    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"]
Oct 30 03:52:39.216369 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:39.216200    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" containerID="cri-o://5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a" gracePeriod=30
Oct 30 03:52:39.216369 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:39.216204    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kube-rbac-proxy" containerID="cri-o://a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536" gracePeriod=30
Oct 30 03:52:39.844801 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:39.844762    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Oct 30 03:52:40.258312 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:52:40.258222    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:52:40.259956 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:40.259934    4377 generic.go:356] "Generic (PLEG): container finished" podID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerID="a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536" exitCode=0
Oct 30 03:52:40.260053 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:40.260001    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerDied","Data":"a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536"}
Oct 30 03:52:43.509548 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:43.509444    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Oct 30 03:52:44.666253 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.666225    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:52:44.829749 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.829710    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e73b3b1-1b43-4d29-a07a-1df947b6e116-proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e73b3b1-1b43-4d29-a07a-1df947b6e116-proxy-tls\") pod \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") "
Oct 30 03:52:44.829979 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.829762    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kserve-provision-location\") pod \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") "
Oct 30 03:52:44.829979 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.829798    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-rbac-proxy-sar-config\") pod \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") "
Oct 30 03:52:44.829979 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.829830    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-cabundle-cert\") pod \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") "
Oct 30 03:52:44.829979 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.829892    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-api-access-f25fw\" (UniqueName: \"kubernetes.io/projected/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-api-access-f25fw\") pod \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\" (UID: \"9e73b3b1-1b43-4d29-a07a-1df947b6e116\") "
Oct 30 03:52:44.830181 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.830136    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kserve-provision-location" pod "9e73b3b1-1b43-4d29-a07a-1df947b6e116" (UID: "9e73b3b1-1b43-4d29-a07a-1df947b6e116"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:52:44.830181 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.830143    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-rbac-proxy-sar-config" pod "9e73b3b1-1b43-4d29-a07a-1df947b6e116" (UID: "9e73b3b1-1b43-4d29-a07a-1df947b6e116"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:52:44.830256 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.830215    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-cabundle-cert" pod "9e73b3b1-1b43-4d29-a07a-1df947b6e116" (UID: "9e73b3b1-1b43-4d29-a07a-1df947b6e116"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:52:44.831956 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.831935    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-api-access-f25fw" pod "9e73b3b1-1b43-4d29-a07a-1df947b6e116" (UID: "9e73b3b1-1b43-4d29-a07a-1df947b6e116"). InnerVolumeSpecName "kube-api-access-f25fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:52:44.832052 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.832011    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e73b3b1-1b43-4d29-a07a-1df947b6e116-proxy-tls" pod "9e73b3b1-1b43-4d29-a07a-1df947b6e116" (UID: "9e73b3b1-1b43-4d29-a07a-1df947b6e116"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:52:44.930652 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.930610    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f25fw\" (UniqueName: \"kubernetes.io/projected/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-api-access-f25fw\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:52:44.930652 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.930643    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e73b3b1-1b43-4d29-a07a-1df947b6e116-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:52:44.930652 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.930655    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:52:44.930874 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.930664    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:52:44.930874 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:44.930673    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9e73b3b1-1b43-4d29-a07a-1df947b6e116-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:52:45.280809 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.280772    4377 generic.go:356] "Generic (PLEG): container finished" podID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerID="5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a" exitCode=0
Oct 30 03:52:45.281000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.280820    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerDied","Data":"5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a"}
Oct 30 03:52:45.281000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.280854    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"
Oct 30 03:52:45.281000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.280909    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr" event={"ID":"9e73b3b1-1b43-4d29-a07a-1df947b6e116","Type":"ContainerDied","Data":"ef0fa6497f8da434c19bfdf0d2efac73b8d87cee1cfd66f337ac88646892fe58"}
Oct 30 03:52:45.281000 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.280939    4377 scope.go:122] "RemoveContainer" containerID="a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536"
Oct 30 03:52:45.289628 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.289609    4377 scope.go:122] "RemoveContainer" containerID="5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a"
Oct 30 03:52:45.296799 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.296777    4377 scope.go:122] "RemoveContainer" containerID="94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81"
Oct 30 03:52:45.303862 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.303846    4377 scope.go:122] "RemoveContainer" containerID="a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536"
Oct 30 03:52:45.304107 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:52:45.304086    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536\": container with ID starting with a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536 not found: ID does not exist" containerID="a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536"
Oct 30 03:52:45.304154 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.304116    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536"} err="failed to get container status \"a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536\": rpc error: code = NotFound desc = could not find container \"a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536\": container with ID starting with a3f05c2743f34860fe75ea2e8003c8f2cd2f6b31135b2a3d07d64420e8d10536 not found: ID does not exist"
Oct 30 03:52:45.304154 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.304133    4377 scope.go:122] "RemoveContainer" containerID="5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a"
Oct 30 03:52:45.304330 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:52:45.304316    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a\": container with ID starting with 5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a not found: ID does not exist" containerID="5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a"
Oct 30 03:52:45.304369 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.304333    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a"} err="failed to get container status \"5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a\": rpc error: code = NotFound desc = could not find container \"5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a\": container with ID starting with 5049fa1e7a06d577f28329e58e267f3c526f97efea84d45a02e98f100bd5868a not found: ID does not exist"
Oct 30 03:52:45.304369 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.304345    4377 scope.go:122] "RemoveContainer" containerID="94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81"
Oct 30 03:52:45.304535 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:52:45.304502    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81\": container with ID starting with 94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81 not found: ID does not exist" containerID="94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81"
Oct 30 03:52:45.304581 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.304537    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81"} err="failed to get container status \"94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81\": rpc error: code = NotFound desc = could not find container \"94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81\": container with ID starting with 94b02320eff1ae44b0f5d9a1b5640723b419dc443562a0b5c01e934f3bc98b81 not found: ID does not exist"
Oct 30 03:52:45.348575 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.348537    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"]
Oct 30 03:52:45.361121 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:45.361088    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-4a54dc1685/isvc-lightgbm-runtime-predictor-6778c96cb9-lncbr"]
Oct 30 03:52:46.513953 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:52:46.513918    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" path="/var/lib/kubelet/pods/9e73b3b1-1b43-4d29-a07a-1df947b6e116/volumes"
Oct 30 03:53:10.260904 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:53:10.260868    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:53:13.162465 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.162426    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"]
Oct 30 03:53:13.162982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.162885    4377 cpu_manager.go:414] "containerMap: removing container" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="storage-initializer"
Oct 30 03:53:13.162982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.162903    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="storage-initializer"
Oct 30 03:53:13.162982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.162918    4377 cpu_manager.go:414] "containerMap: removing container" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container"
Oct 30 03:53:13.162982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.162925    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container"
Oct 30 03:53:13.162982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.162969    4377 cpu_manager.go:414] "containerMap: removing container" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kube-rbac-proxy"
Oct 30 03:53:13.162982 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.162976    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kube-rbac-proxy"
Oct 30 03:53:13.163349 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.163146    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kserve-container"
Oct 30 03:53:13.163349 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.163169    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e73b3b1-1b43-4d29-a07a-1df947b6e116" containerName="kube-rbac-proxy"
Oct 30 03:53:13.166271 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.166249    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.181473 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.181453    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"storage-config\""
Oct 30 03:53:13.181625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.181607    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Oct 30 03:53:13.181679 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.181608    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:53:13.194085 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.194062    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:53:13.194197 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.194069    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:53:13.194197 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.194092    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Oct 30 03:53:13.194197 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.194103    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:53:13.195512 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.195491    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"]
Oct 30 03:53:13.252358 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.252330    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.252530 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.252364    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.252530 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.252385    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.252530 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.252505    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djf42\" (UniqueName: \"kubernetes.io/projected/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-api-access-djf42\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.252646 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.252580    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.353697 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.353662    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.353697 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.353704    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.353947 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.353728    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.353947 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.353789    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djf42\" (UniqueName: \"kubernetes.io/projected/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-api-access-djf42\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.353947 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.353851    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.354187 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.354157    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.354502 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.354477    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.354502 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.354489    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.356382 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.356358    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.377499 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.377469    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djf42\" (UniqueName: \"kubernetes.io/projected/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-api-access-djf42\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.482643 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.482545    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:13.793418 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:13.793392    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"]
Oct 30 03:53:13.795804 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:53:13.795778    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf157d1ca_3b06_4f95_bfa2_0033ac0d80aa.slice/crio-c43b3f40d462b9c044e1565baf005804676a6f7bd45c355e2362097e09e2e498 WatchSource:0}: Error finding container c43b3f40d462b9c044e1565baf005804676a6f7bd45c355e2362097e09e2e498: Status 404 returned error can't find the container with id c43b3f40d462b9c044e1565baf005804676a6f7bd45c355e2362097e09e2e498
Oct 30 03:53:14.395143 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:14.395105    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerStarted","Data":"f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7"}
Oct 30 03:53:14.395639 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:14.395209    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerStarted","Data":"c43b3f40d462b9c044e1565baf005804676a6f7bd45c355e2362097e09e2e498"}
Oct 30 03:53:15.400836 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:15.400799    4377 generic.go:356] "Generic (PLEG): container finished" podID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerID="f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7" exitCode=0
Oct 30 03:53:15.401261 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:15.400889    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerDied","Data":"f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7"}
Oct 30 03:53:16.406395 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:16.406355    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerStarted","Data":"c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12"}
Oct 30 03:53:16.406853 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:16.406430    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerStarted","Data":"35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386"}
Oct 30 03:53:16.406853 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:16.406670    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:16.406853 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:16.406814    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:16.408151 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:16.408128    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:53:17.410065 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:17.410026    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:53:22.414746 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:22.414713    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:53:22.415249 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:22.415194    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:53:22.473446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:22.473392    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podStartSLOduration=8.432797936 podStartE2EDuration="9.473376882s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.040578946s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:53:16.468987998 +0000 UTC m=+3496.061316803" watchObservedRunningTime="2026-10-30 03:53:22.473376882 +0000 UTC m=+3502.065705687"
Oct 30 03:53:32.415280 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:32.415240    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:53:40.258327 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:53:40.258292    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:53:42.416193 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:42.416157    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:53:52.415484 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:53:52.415447    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:54:02.415739 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:02.415699    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:54:10.258495 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:54:10.258456    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:54:12.416138 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:12.416101    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:54:22.415532 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:22.415481    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Oct 30 03:54:32.416031 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:32.415999    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:54:33.142399 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:33.142359    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"]
Oct 30 03:54:33.142667 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:33.142617    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container" containerID="cri-o://35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386" gracePeriod=30
Oct 30 03:54:33.142784 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:33.142666    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kube-rbac-proxy" containerID="cri-o://c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12" gracePeriod=30
Oct 30 03:54:33.703507 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:33.703468    4377 generic.go:356] "Generic (PLEG): container finished" podID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerID="c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12" exitCode=0
Oct 30 03:54:33.703905 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:33.703551    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerDied","Data":"c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12"}
Oct 30 03:54:37.410568 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:37.410495    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Oct 30 03:54:37.986482 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:37.986457    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:54:38.161386 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161297    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-rbac-proxy-sar-config\") pod \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") "
Oct 30 03:54:38.161386 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161376    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kserve-provision-location\") pod \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") "
Oct 30 03:54:38.161596 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161547    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-cabundle-cert\") pod \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") "
Oct 30 03:54:38.161634 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161606    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-api-access-djf42\" (UniqueName: \"kubernetes.io/projected/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-api-access-djf42\") pod \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") "
Oct 30 03:54:38.161668 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161648    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-proxy-tls\") pod \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\" (UID: \"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa\") "
Oct 30 03:54:38.161728 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161705    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kserve-provision-location" pod "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" (UID: "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:54:38.161781 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161743    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-rbac-proxy-sar-config" pod "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" (UID: "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:54:38.161929 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161909    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:54:38.162006 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161928    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-cabundle-cert" pod "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" (UID: "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:54:38.162006 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.161934    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:54:38.163683 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.163659    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-proxy-tls" pod "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" (UID: "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:54:38.163772 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.163708    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-api-access-djf42" pod "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" (UID: "f157d1ca-3b06-4f95-bfa2-0033ac0d80aa"). InnerVolumeSpecName "kube-api-access-djf42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:54:38.262799 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.262763    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:54:38.262799 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.262794    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djf42\" (UniqueName: \"kubernetes.io/projected/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-kube-api-access-djf42\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:54:38.262799 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.262806    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:54:38.727179 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.727144    4377 generic.go:356] "Generic (PLEG): container finished" podID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerID="35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386" exitCode=0
Oct 30 03:54:38.727609 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.727206    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"
Oct 30 03:54:38.727609 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.727234    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerDied","Data":"35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386"}
Oct 30 03:54:38.727609 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.727294    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f" event={"ID":"f157d1ca-3b06-4f95-bfa2-0033ac0d80aa","Type":"ContainerDied","Data":"c43b3f40d462b9c044e1565baf005804676a6f7bd45c355e2362097e09e2e498"}
Oct 30 03:54:38.727609 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.727321    4377 scope.go:122] "RemoveContainer" containerID="c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12"
Oct 30 03:54:38.737258 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.737236    4377 scope.go:122] "RemoveContainer" containerID="35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386"
Oct 30 03:54:38.744882 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.744865    4377 scope.go:122] "RemoveContainer" containerID="f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7"
Oct 30 03:54:38.751843 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.751828    4377 scope.go:122] "RemoveContainer" containerID="c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12"
Oct 30 03:54:38.752095 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:54:38.752076    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12\": container with ID starting with c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12 not found: ID does not exist" containerID="c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12"
Oct 30 03:54:38.752157 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.752107    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12"} err="failed to get container status \"c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12\": rpc error: code = NotFound desc = could not find container \"c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12\": container with ID starting with c91911286b7df50b478f054eb2ff5e5ac03086920a5cb25a9ba384ffb140cf12 not found: ID does not exist"
Oct 30 03:54:38.752157 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.752125    4377 scope.go:122] "RemoveContainer" containerID="35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386"
Oct 30 03:54:38.752376 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:54:38.752357    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386\": container with ID starting with 35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386 not found: ID does not exist" containerID="35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386"
Oct 30 03:54:38.752423 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.752382    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386"} err="failed to get container status \"35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386\": rpc error: code = NotFound desc = could not find container \"35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386\": container with ID starting with 35c1a290100a0701d343e2fac90174e2c3a5baaea30cff5b05f137c3134b7386 not found: ID does not exist"
Oct 30 03:54:38.752423 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.752399    4377 scope.go:122] "RemoveContainer" containerID="f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7"
Oct 30 03:54:38.752688 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:54:38.752668    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7\": container with ID starting with f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7 not found: ID does not exist" containerID="f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7"
Oct 30 03:54:38.752749 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.752692    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7"} err="failed to get container status \"f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7\": rpc error: code = NotFound desc = could not find container \"f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7\": container with ID starting with f9ab273f52eedecf8b55b772c7d94cb71b43b877681c396f3b99ab8e2bec85b7 not found: ID does not exist"
Oct 30 03:54:38.795423 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.795392    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"]
Oct 30 03:54:38.808193 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:38.808169    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-global-pass-predictor-5775bf8ff8-xtm7f"]
Oct 30 03:54:40.258269 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:54:40.258232    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:54:40.514651 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:40.514574    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" path="/var/lib/kubelet/pods/f157d1ca-3b06-4f95-bfa2-0033ac0d80aa/volumes"
Oct 30 03:54:46.031669 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.031633    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"]
Oct 30 03:54:46.032171 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032093    4377 cpu_manager.go:414] "containerMap: removing container" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="storage-initializer"
Oct 30 03:54:46.032171 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032113    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="storage-initializer"
Oct 30 03:54:46.032171 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032136    4377 cpu_manager.go:414] "containerMap: removing container" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container"
Oct 30 03:54:46.032171 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032145    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container"
Oct 30 03:54:46.032171 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032158    4377 cpu_manager.go:414] "containerMap: removing container" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kube-rbac-proxy"
Oct 30 03:54:46.032171 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032164    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kube-rbac-proxy"
Oct 30 03:54:46.032389 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032227    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kserve-container"
Oct 30 03:54:46.032389 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.032234    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="f157d1ca-3b06-4f95-bfa2-0033ac0d80aa" containerName="kube-rbac-proxy"
Oct 30 03:54:46.036689 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.036672    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.053832 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.053808    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:54:46.053952 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.053878    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Oct 30 03:54:46.066684 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.066659    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:54:46.066684 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.066670    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"storage-config\""
Oct 30 03:54:46.066684 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.066660    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Oct 30 03:54:46.068146 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.068128    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"]
Oct 30 03:54:46.130751 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.130709    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4z8\" (UniqueName: \"kubernetes.io/projected/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-api-access-lt4z8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.130751 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.130752    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.130958 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.130842    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a957699e-593c-4ca3-8c34-1fef77a8d73f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.130958 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.130867    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.130958 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.130894    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a957699e-593c-4ca3-8c34-1fef77a8d73f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.154019 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.153990    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:54:46.154110 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.154072    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:54:46.231536 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.231486    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4z8\" (UniqueName: \"kubernetes.io/projected/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-api-access-lt4z8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.231720 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.231557    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.231720 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.231590    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a957699e-593c-4ca3-8c34-1fef77a8d73f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.231720 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.231610    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.231720 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.231647    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a957699e-593c-4ca3-8c34-1fef77a8d73f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.232145 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.232116    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a957699e-593c-4ca3-8c34-1fef77a8d73f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.232408 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.232386    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.232473 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.232386    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.234042 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.234024    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a957699e-593c-4ca3-8c34-1fef77a8d73f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.255340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.255314    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4z8\" (UniqueName: \"kubernetes.io/projected/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-api-access-lt4z8\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.353627 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.353511    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:46.765648 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:46.765562    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"]
Oct 30 03:54:46.768882 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:54:46.768851    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda957699e_593c_4ca3_8c34_1fef77a8d73f.slice/crio-33e4c7910146a99bf8353c879f0b1a94305fc844df82990437592297c6188b2b WatchSource:0}: Error finding container 33e4c7910146a99bf8353c879f0b1a94305fc844df82990437592297c6188b2b: Status 404 returned error can't find the container with id 33e4c7910146a99bf8353c879f0b1a94305fc844df82990437592297c6188b2b
Oct 30 03:54:47.764138 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:47.764102    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerStarted","Data":"72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30"}
Oct 30 03:54:47.764552 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:47.764186    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerStarted","Data":"33e4c7910146a99bf8353c879f0b1a94305fc844df82990437592297c6188b2b"}
Oct 30 03:54:48.769351 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:48.769313    4377 generic.go:356] "Generic (PLEG): container finished" podID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerID="72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30" exitCode=0
Oct 30 03:54:48.769889 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:48.769405    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerDied","Data":"72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30"}
Oct 30 03:54:49.776975 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:49.776937    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerStarted","Data":"10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa"}
Oct 30 03:54:49.777414 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:49.777006    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerStarted","Data":"f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b"}
Oct 30 03:54:49.777414 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:49.777207    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:49.840512 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:49.839957    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podStartSLOduration=2.7887153959999997 podStartE2EDuration="3.839935119s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.051219723s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:54:49.83933842 +0000 UTC m=+3589.431667244" watchObservedRunningTime="2026-10-30 03:54:49.839935119 +0000 UTC m=+3589.432263955"
Oct 30 03:54:50.780566 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:50.780535    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:50.782026 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:50.781990    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:54:51.784629 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:51.784578    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:54:56.789854 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:56.789812    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:54:56.790409 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:54:56.790382    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:55:06.790327 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:55:06.790284    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:55:10.258647 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:55:10.258608    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:55:16.790456 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:55:16.790413    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:55:26.790460 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:55:26.790423    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:55:36.790473 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:55:36.790436    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:55:40.259591 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:55:40.259550    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:55:46.790641 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:55:46.790555    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:55:56.790542 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:55:56.790489    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:56:02.514914 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:02.514861    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Oct 30 03:56:10.259577 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:56:10.259540    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:56:12.514336 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:12.514306    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:56:16.139621 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:16.139587    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"]
Oct 30 03:56:16.140090 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:16.139803    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container" containerID="cri-o://f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b" gracePeriod=30
Oct 30 03:56:16.140090 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:16.139840    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kube-rbac-proxy" containerID="cri-o://10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa" gracePeriod=30
Oct 30 03:56:16.785801 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:16.785762    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Oct 30 03:56:17.129308 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:17.129229    4377 generic.go:356] "Generic (PLEG): container finished" podID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerID="10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa" exitCode=0
Oct 30 03:56:17.129452 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:17.129301    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerDied","Data":"10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa"}
Oct 30 03:56:20.913597 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:56:20.913561    4377 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda957699e_593c_4ca3_8c34_1fef77a8d73f.slice/crio-f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda957699e_593c_4ca3_8c34_1fef77a8d73f.slice/crio-conmon-f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 03:56:21.079757 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.079732    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:56:21.147227 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.147190    4377 generic.go:356] "Generic (PLEG): container finished" podID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerID="f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b" exitCode=0
Oct 30 03:56:21.147412 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.147236    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerDied","Data":"f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b"}
Oct 30 03:56:21.147412 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.147293    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2" event={"ID":"a957699e-593c-4ca3-8c34-1fef77a8d73f","Type":"ContainerDied","Data":"33e4c7910146a99bf8353c879f0b1a94305fc844df82990437592297c6188b2b"}
Oct 30 03:56:21.147412 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.147316    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"
Oct 30 03:56:21.147412 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.147323    4377 scope.go:122] "RemoveContainer" containerID="10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa"
Oct 30 03:56:21.155425 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.155406    4377 scope.go:122] "RemoveContainer" containerID="f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b"
Oct 30 03:56:21.162845 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.162825    4377 scope.go:122] "RemoveContainer" containerID="72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30"
Oct 30 03:56:21.170007 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.169988    4377 scope.go:122] "RemoveContainer" containerID="10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa"
Oct 30 03:56:21.170301 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:56:21.170283    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa\": container with ID starting with 10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa not found: ID does not exist" containerID="10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa"
Oct 30 03:56:21.170383 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.170315    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa"} err="failed to get container status \"10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa\": rpc error: code = NotFound desc = could not find container \"10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa\": container with ID starting with 10800bae40162ce352147b982509afdd41e49f0ca6bcbc39e97e7bd694695bfa not found: ID does not exist"
Oct 30 03:56:21.170383 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.170339    4377 scope.go:122] "RemoveContainer" containerID="f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b"
Oct 30 03:56:21.170633 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:56:21.170612    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b\": container with ID starting with f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b not found: ID does not exist" containerID="f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b"
Oct 30 03:56:21.170681 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.170643    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b"} err="failed to get container status \"f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b\": rpc error: code = NotFound desc = could not find container \"f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b\": container with ID starting with f5d098ea3ba0fbbf0feb8fa6d3530daa6137ce85a4187573561100376262257b not found: ID does not exist"
Oct 30 03:56:21.170681 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.170659    4377 scope.go:122] "RemoveContainer" containerID="72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30"
Oct 30 03:56:21.170881 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:56:21.170863    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30\": container with ID starting with 72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30 not found: ID does not exist" containerID="72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30"
Oct 30 03:56:21.170938 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.170892    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30"} err="failed to get container status \"72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30\": rpc error: code = NotFound desc = could not find container \"72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30\": container with ID starting with 72f24e50145d55dbd4a5aaf47fb7c7602f219f4787bae5a3940b0abd85cd3e30 not found: ID does not exist"
Oct 30 03:56:21.212184 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212160    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a957699e-593c-4ca3-8c34-1fef77a8d73f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a957699e-593c-4ca3-8c34-1fef77a8d73f-kserve-provision-location\") pod \"a957699e-593c-4ca3-8c34-1fef77a8d73f\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") "
Oct 30 03:56:21.212291 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212221    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-api-access-lt4z8\" (UniqueName: \"kubernetes.io/projected/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-api-access-lt4z8\") pod \"a957699e-593c-4ca3-8c34-1fef77a8d73f\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") "
Oct 30 03:56:21.212340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212303    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-cabundle-cert\") pod \"a957699e-593c-4ca3-8c34-1fef77a8d73f\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") "
Oct 30 03:56:21.212373 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212344    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a957699e-593c-4ca3-8c34-1fef77a8d73f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/a957699e-593c-4ca3-8c34-1fef77a8d73f-proxy-tls\") pod \"a957699e-593c-4ca3-8c34-1fef77a8d73f\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") "
Oct 30 03:56:21.212446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212395    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-rbac-proxy-sar-config\") pod \"a957699e-593c-4ca3-8c34-1fef77a8d73f\" (UID: \"a957699e-593c-4ca3-8c34-1fef77a8d73f\") "
Oct 30 03:56:21.212557 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212514    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a957699e-593c-4ca3-8c34-1fef77a8d73f-kserve-provision-location" pod "a957699e-593c-4ca3-8c34-1fef77a8d73f" (UID: "a957699e-593c-4ca3-8c34-1fef77a8d73f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:56:21.212640 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212627    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a957699e-593c-4ca3-8c34-1fef77a8d73f-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:56:21.212804 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212680    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-cabundle-cert" pod "a957699e-593c-4ca3-8c34-1fef77a8d73f" (UID: "a957699e-593c-4ca3-8c34-1fef77a8d73f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:21.212851 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.212830    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-rbac-proxy-sar-config" pod "a957699e-593c-4ca3-8c34-1fef77a8d73f" (UID: "a957699e-593c-4ca3-8c34-1fef77a8d73f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:56:21.214340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.214317    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-api-access-lt4z8" pod "a957699e-593c-4ca3-8c34-1fef77a8d73f" (UID: "a957699e-593c-4ca3-8c34-1fef77a8d73f"). InnerVolumeSpecName "kube-api-access-lt4z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:56:21.214340 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.214320    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a957699e-593c-4ca3-8c34-1fef77a8d73f-proxy-tls" pod "a957699e-593c-4ca3-8c34-1fef77a8d73f" (UID: "a957699e-593c-4ca3-8c34-1fef77a8d73f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:56:21.313002 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.312967    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt4z8\" (UniqueName: \"kubernetes.io/projected/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-api-access-lt4z8\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:56:21.313002 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.313001    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:56:21.313209 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.313018    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a957699e-593c-4ca3-8c34-1fef77a8d73f-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:56:21.313209 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.313030    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a957699e-593c-4ca3-8c34-1fef77a8d73f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:56:21.497928 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.497894    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"]
Oct 30 03:56:21.503234 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:21.503211    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-custom-pass-predictor-86995f8f78-llzs2"]
Oct 30 03:56:22.514480 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:22.514447    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" path="/var/lib/kubelet/pods/a957699e-593c-4ca3-8c34-1fef77a8d73f/volumes"
Oct 30 03:56:35.104758 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.104722    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"]
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105036    4377 cpu_manager.go:414] "containerMap: removing container" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container"
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105048    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container"
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105061    4377 cpu_manager.go:414] "containerMap: removing container" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kube-rbac-proxy"
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105066    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kube-rbac-proxy"
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105074    4377 cpu_manager.go:414] "containerMap: removing container" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="storage-initializer"
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105080    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="storage-initializer"
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105142    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kserve-container"
Oct 30 03:56:35.105322 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.105149    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="a957699e-593c-4ca3-8c34-1fef77a8d73f" containerName="kube-rbac-proxy"
Oct 30 03:56:35.110166 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.110144    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.127757 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.127730    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:56:35.127873 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.127754    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Oct 30 03:56:35.140211 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.140185    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:56:35.140382 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.140195    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:56:35.140506 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.140195    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"storage-config\""
Oct 30 03:56:35.140506 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.140227    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:56:35.140506 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.140234    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Oct 30 03:56:35.141679 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.141657    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"]
Oct 30 03:56:35.231848 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.231814    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.232010 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.231854    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f884v\" (UniqueName: \"kubernetes.io/projected/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-api-access-f884v\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.232010 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.231940    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.232010 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.231971    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.232010 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.231997    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.333158 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333126    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.333310 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333170    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f884v\" (UniqueName: \"kubernetes.io/projected/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-api-access-f884v\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.333310 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333223    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.333310 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333249    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.333310 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333283    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.333746 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333719    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.333895 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333868    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.334006 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.333896    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.335596 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.335580    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.357278 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.357213    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f884v\" (UniqueName: \"kubernetes.io/projected/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-api-access-f884v\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.427618 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.427578    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:35.739931 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:35.739902    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"]
Oct 30 03:56:36.210549 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:36.210499    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerStarted","Data":"4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab"}
Oct 30 03:56:36.210942 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:36.210571    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerStarted","Data":"a8ac72828c5fddbde5e6cb24f5b6f82ec44c9283fadd1e1bdd23da16218514b9"}
Oct 30 03:56:37.217139 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:37.217054    4377 generic.go:356] "Generic (PLEG): container finished" podID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerID="4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab" exitCode=0
Oct 30 03:56:37.217139 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:37.217129    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerDied","Data":"4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab"}
Oct 30 03:56:38.222816 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:38.222778    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerStarted","Data":"714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a"}
Oct 30 03:56:38.223213 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:38.222838    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerStarted","Data":"778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8"}
Oct 30 03:56:38.223213 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:38.222868    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:38.282583 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:38.282511    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podStartSLOduration=2.243055394 podStartE2EDuration="3.28249445s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.039439056s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:56:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:56:38.281612159 +0000 UTC m=+3697.873940965" watchObservedRunningTime="2026-10-30 03:56:38.28249445 +0000 UTC m=+3697.874823256"
Oct 30 03:56:39.226989 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:39.226952    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:39.228165 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:39.228133    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:56:40.231131 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:40.231090    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:56:40.259185 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:56:40.259152    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:56:45.235855 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:45.235824    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:56:45.236327 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:45.236302    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:56:55.237231 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:56:55.237188    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:57:05.237248 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:57:05.237209    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:57:10.258817 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:57:10.258778    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:57:15.236863 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:57:15.236774    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:57:25.236631 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:57:25.236591    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:57:35.236868 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:57:35.236824    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:57:40.258483 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:57:40.258442    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:57:45.237242 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:57:45.237196    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:57:55.238154 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:57:55.238124    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:58:05.194041 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:05.194004    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"]
Oct 30 03:58:05.194478 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:05.194248    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" containerID="cri-o://778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8" gracePeriod=30
Oct 30 03:58:05.194478 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:05.194302    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kube-rbac-proxy" containerID="cri-o://714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a" gracePeriod=30
Oct 30 03:58:05.231435 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:05.231404    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Oct 30 03:58:05.236733 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:05.236710    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Oct 30 03:58:05.566643 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:05.566607    4377 generic.go:356] "Generic (PLEG): container finished" podID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerID="714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a" exitCode=0
Oct 30 03:58:05.566814 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:05.566680    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerDied","Data":"714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a"}
Oct 30 03:58:06.508974 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.508929    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"]
Oct 30 03:58:06.512650 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.512625    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.529027 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.529008    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Oct 30 03:58:06.529228 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.529209    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Oct 30 03:58:06.544155 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.544128    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"]
Oct 30 03:58:06.578463 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.578434    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pl4m\" (UniqueName: \"kubernetes.io/projected/e057af9d-4831-420c-90e4-80732e8690e0-kube-api-access-8pl4m\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.578625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.578492    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e057af9d-4831-420c-90e4-80732e8690e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.578625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.578590    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e057af9d-4831-420c-90e4-80732e8690e0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.578715 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.578658    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e057af9d-4831-420c-90e4-80732e8690e0-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.679168 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.679133    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e057af9d-4831-420c-90e4-80732e8690e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.679168 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.679171    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e057af9d-4831-420c-90e4-80732e8690e0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.679422 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.679216    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e057af9d-4831-420c-90e4-80732e8690e0-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.679422 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.679240    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pl4m\" (UniqueName: \"kubernetes.io/projected/e057af9d-4831-420c-90e4-80732e8690e0-kube-api-access-8pl4m\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.679706 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.679683    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e057af9d-4831-420c-90e4-80732e8690e0-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.679847 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.679826    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e057af9d-4831-420c-90e4-80732e8690e0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.681615 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.681599    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e057af9d-4831-420c-90e4-80732e8690e0-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.703456 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.703430    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pl4m\" (UniqueName: \"kubernetes.io/projected/e057af9d-4831-420c-90e4-80732e8690e0-kube-api-access-8pl4m\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") " pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:06.830140 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:06.830106    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:07.141304 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:07.141276    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"]
Oct 30 03:58:07.143357 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:58:07.143329    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode057af9d_4831_420c_90e4_80732e8690e0.slice/crio-7d43161d4830d16b7a30e4ce08689752bfd39d288bcc5a06c0a3aca7606255ca WatchSource:0}: Error finding container 7d43161d4830d16b7a30e4ce08689752bfd39d288bcc5a06c0a3aca7606255ca: Status 404 returned error can't find the container with id 7d43161d4830d16b7a30e4ce08689752bfd39d288bcc5a06c0a3aca7606255ca
Oct 30 03:58:07.576418 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:07.576383    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66" event={"ID":"e057af9d-4831-420c-90e4-80732e8690e0","Type":"ContainerStarted","Data":"1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8"}
Oct 30 03:58:07.576817 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:07.576454    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66" event={"ID":"e057af9d-4831-420c-90e4-80732e8690e0","Type":"ContainerStarted","Data":"7d43161d4830d16b7a30e4ce08689752bfd39d288bcc5a06c0a3aca7606255ca"}
Oct 30 03:58:10.139744 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.139720    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:58:10.212195 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212116    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-api-access-f884v\" (UniqueName: \"kubernetes.io/projected/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-api-access-f884v\") pod \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") "
Oct 30 03:58:10.212195 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212179    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-proxy-tls\") pod \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") "
Oct 30 03:58:10.212414 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212217    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kserve-provision-location\") pod \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") "
Oct 30 03:58:10.212414 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212264    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-cabundle-cert\") pod \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") "
Oct 30 03:58:10.212414 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212295    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-rbac-proxy-sar-config\") pod \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\" (UID: \"6ea8d8d7-7b62-4e19-8461-336b4802a2fa\") "
Oct 30 03:58:10.212627 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212598    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kserve-provision-location" pod "6ea8d8d7-7b62-4e19-8461-336b4802a2fa" (UID: "6ea8d8d7-7b62-4e19-8461-336b4802a2fa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:10.212691 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212649    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-cabundle-cert" pod "6ea8d8d7-7b62-4e19-8461-336b4802a2fa" (UID: "6ea8d8d7-7b62-4e19-8461-336b4802a2fa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:10.212748 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.212719    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-rbac-proxy-sar-config" pod "6ea8d8d7-7b62-4e19-8461-336b4802a2fa" (UID: "6ea8d8d7-7b62-4e19-8461-336b4802a2fa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:10.214390 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.214358    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-proxy-tls" pod "6ea8d8d7-7b62-4e19-8461-336b4802a2fa" (UID: "6ea8d8d7-7b62-4e19-8461-336b4802a2fa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:10.214477 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.214358    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-api-access-f884v" pod "6ea8d8d7-7b62-4e19-8461-336b4802a2fa" (UID: "6ea8d8d7-7b62-4e19-8461-336b4802a2fa"). InnerVolumeSpecName "kube-api-access-f884v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:10.257615 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:58:10.257580    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:58:10.313166 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.313138    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f884v\" (UniqueName: \"kubernetes.io/projected/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-api-access-f884v\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:10.313166 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.313164    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:10.313360 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.313175    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:10.313360 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.313184    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:10.313360 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.313192    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ea8d8d7-7b62-4e19-8461-336b4802a2fa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:10.590058 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.590026    4377 generic.go:356] "Generic (PLEG): container finished" podID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerID="778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8" exitCode=0
Oct 30 03:58:10.590247 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.590091    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"
Oct 30 03:58:10.590247 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.590087    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerDied","Data":"778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8"}
Oct 30 03:58:10.590247 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.590171    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv" event={"ID":"6ea8d8d7-7b62-4e19-8461-336b4802a2fa","Type":"ContainerDied","Data":"a8ac72828c5fddbde5e6cb24f5b6f82ec44c9283fadd1e1bdd23da16218514b9"}
Oct 30 03:58:10.590247 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.590215    4377 scope.go:122] "RemoveContainer" containerID="714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a"
Oct 30 03:58:10.591796 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.591779    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66_e057af9d-4831-420c-90e4-80732e8690e0/storage-initializer/0.log"
Oct 30 03:58:10.591902 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.591809    4377 generic.go:356] "Generic (PLEG): container finished" podID="e057af9d-4831-420c-90e4-80732e8690e0" containerID="1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8" exitCode=1
Oct 30 03:58:10.591902 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.591847    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66" event={"ID":"e057af9d-4831-420c-90e4-80732e8690e0","Type":"ContainerDied","Data":"1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8"}
Oct 30 03:58:10.600893 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.600876    4377 scope.go:122] "RemoveContainer" containerID="778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8"
Oct 30 03:58:10.615130 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.615112    4377 scope.go:122] "RemoveContainer" containerID="4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab"
Oct 30 03:58:10.623818 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.623797    4377 scope.go:122] "RemoveContainer" containerID="714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a"
Oct 30 03:58:10.624072 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:58:10.624056    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a\": container with ID starting with 714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a not found: ID does not exist" containerID="714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a"
Oct 30 03:58:10.624121 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.624081    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a"} err="failed to get container status \"714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a\": rpc error: code = NotFound desc = could not find container \"714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a\": container with ID starting with 714ebc240d0b896ff78d93e209fae580714f7a63ed24187bfa1c6489210a687a not found: ID does not exist"
Oct 30 03:58:10.624121 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.624095    4377 scope.go:122] "RemoveContainer" containerID="778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8"
Oct 30 03:58:10.624329 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:58:10.624314    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8\": container with ID starting with 778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8 not found: ID does not exist" containerID="778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8"
Oct 30 03:58:10.624383 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.624332    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8"} err="failed to get container status \"778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8\": rpc error: code = NotFound desc = could not find container \"778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8\": container with ID starting with 778ac409a19437a1510cc49bdafb89f1665bf5db6f0cf045bb5ab199da16d6f8 not found: ID does not exist"
Oct 30 03:58:10.624383 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.624345    4377 scope.go:122] "RemoveContainer" containerID="4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab"
Oct 30 03:58:10.624652 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:58:10.624635    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab\": container with ID starting with 4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab not found: ID does not exist" containerID="4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab"
Oct 30 03:58:10.624730 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.624653    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab"} err="failed to get container status \"4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab\": rpc error: code = NotFound desc = could not find container \"4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab\": container with ID starting with 4f33752551a606c2cbd5b2c57c4587b207298451f5ceba9747f11e42322d74ab not found: ID does not exist"
Oct 30 03:58:10.691536 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.691497    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"]
Oct 30 03:58:10.695585 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:10.695555    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-pass-predictor-54997fffc5-p9hmv"]
Oct 30 03:58:11.599485 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:11.599457    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66_e057af9d-4831-420c-90e4-80732e8690e0/storage-initializer/0.log"
Oct 30 03:58:11.599900 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:11.599575    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66" event={"ID":"e057af9d-4831-420c-90e4-80732e8690e0","Type":"ContainerStarted","Data":"f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e"}
Oct 30 03:58:12.514730 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:12.514701    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" path="/var/lib/kubelet/pods/6ea8d8d7-7b62-4e19-8461-336b4802a2fa/volumes"
Oct 30 03:58:16.464799 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.464769    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"]
Oct 30 03:58:16.465354 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.464977    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66" podUID="e057af9d-4831-420c-90e4-80732e8690e0" containerName="storage-initializer" containerID="cri-o://f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e" gracePeriod=30
Oct 30 03:58:16.603802 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.603780    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66_e057af9d-4831-420c-90e4-80732e8690e0/storage-initializer/1.log"
Oct 30 03:58:16.604208 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.604192    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66_e057af9d-4831-420c-90e4-80732e8690e0/storage-initializer/0.log"
Oct 30 03:58:16.606574 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.606557    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:16.620828 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.620802    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66_e057af9d-4831-420c-90e4-80732e8690e0/storage-initializer/1.log"
Oct 30 03:58:16.621177 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.621164    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-773c71d3d4_isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66_e057af9d-4831-420c-90e4-80732e8690e0/storage-initializer/0.log"
Oct 30 03:58:16.621254 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.621193    4377 generic.go:356] "Generic (PLEG): container finished" podID="e057af9d-4831-420c-90e4-80732e8690e0" containerID="f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e" exitCode=1
Oct 30 03:58:16.621307 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.621266    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66" event={"ID":"e057af9d-4831-420c-90e4-80732e8690e0","Type":"ContainerDied","Data":"f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e"}
Oct 30 03:58:16.621353 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.621316    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66" event={"ID":"e057af9d-4831-420c-90e4-80732e8690e0","Type":"ContainerDied","Data":"7d43161d4830d16b7a30e4ce08689752bfd39d288bcc5a06c0a3aca7606255ca"}
Oct 30 03:58:16.621353 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.621344    4377 scope.go:122] "RemoveContainer" containerID="f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e"
Oct 30 03:58:16.621419 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.621274    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"
Oct 30 03:58:16.632910 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.632887    4377 scope.go:122] "RemoveContainer" containerID="1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8"
Oct 30 03:58:16.640060 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.640044    4377 scope.go:122] "RemoveContainer" containerID="f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e"
Oct 30 03:58:16.640326 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:58:16.640303    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e\": container with ID starting with f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e not found: ID does not exist" containerID="f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e"
Oct 30 03:58:16.640370 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.640335    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e"} err="failed to get container status \"f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e\": rpc error: code = NotFound desc = could not find container \"f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e\": container with ID starting with f1ca28c1a8ca865ecd13564593b5a8bf2467b0e0ff9267e52ed242a779f7f88e not found: ID does not exist"
Oct 30 03:58:16.640370 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.640350    4377 scope.go:122] "RemoveContainer" containerID="1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8"
Oct 30 03:58:16.640577 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:58:16.640559    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8\": container with ID starting with 1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8 not found: ID does not exist" containerID="1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8"
Oct 30 03:58:16.640625 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.640584    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8"} err="failed to get container status \"1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8\": rpc error: code = NotFound desc = could not find container \"1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8\": container with ID starting with 1384f3c487823c869073450bd4e56510565e2feb6dafd27938145c3c437fcce8 not found: ID does not exist"
Oct 30 03:58:16.767144 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.767062    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e057af9d-4831-420c-90e4-80732e8690e0-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e057af9d-4831-420c-90e4-80732e8690e0-kube-rbac-proxy-sar-config\") pod \"e057af9d-4831-420c-90e4-80732e8690e0\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") "
Oct 30 03:58:16.767144 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.767136    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e057af9d-4831-420c-90e4-80732e8690e0-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e057af9d-4831-420c-90e4-80732e8690e0-kserve-provision-location\") pod \"e057af9d-4831-420c-90e4-80732e8690e0\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") "
Oct 30 03:58:16.767344 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.767160    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e057af9d-4831-420c-90e4-80732e8690e0-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e057af9d-4831-420c-90e4-80732e8690e0-proxy-tls\") pod \"e057af9d-4831-420c-90e4-80732e8690e0\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") "
Oct 30 03:58:16.767344 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.767208    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e057af9d-4831-420c-90e4-80732e8690e0-kube-api-access-8pl4m\" (UniqueName: \"kubernetes.io/projected/e057af9d-4831-420c-90e4-80732e8690e0-kube-api-access-8pl4m\") pod \"e057af9d-4831-420c-90e4-80732e8690e0\" (UID: \"e057af9d-4831-420c-90e4-80732e8690e0\") "
Oct 30 03:58:16.767434 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.767407    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e057af9d-4831-420c-90e4-80732e8690e0-kserve-provision-location" pod "e057af9d-4831-420c-90e4-80732e8690e0" (UID: "e057af9d-4831-420c-90e4-80732e8690e0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:58:16.767501 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.767486    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e057af9d-4831-420c-90e4-80732e8690e0-kube-rbac-proxy-sar-config" pod "e057af9d-4831-420c-90e4-80732e8690e0" (UID: "e057af9d-4831-420c-90e4-80732e8690e0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:58:16.769288 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.769266    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e057af9d-4831-420c-90e4-80732e8690e0-kube-api-access-8pl4m" pod "e057af9d-4831-420c-90e4-80732e8690e0" (UID: "e057af9d-4831-420c-90e4-80732e8690e0"). InnerVolumeSpecName "kube-api-access-8pl4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:58:16.769444 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.769429    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e057af9d-4831-420c-90e4-80732e8690e0-proxy-tls" pod "e057af9d-4831-420c-90e4-80732e8690e0" (UID: "e057af9d-4831-420c-90e4-80732e8690e0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:58:16.868626 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.868590    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e057af9d-4831-420c-90e4-80732e8690e0-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:16.868626 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.868618    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e057af9d-4831-420c-90e4-80732e8690e0-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:16.868626 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.868628    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pl4m\" (UniqueName: \"kubernetes.io/projected/e057af9d-4831-420c-90e4-80732e8690e0-kube-api-access-8pl4m\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:16.868626 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:16.868637    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e057af9d-4831-420c-90e4-80732e8690e0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:58:17.000471 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:17.000441    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"]
Oct 30 03:58:17.004679 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:17.004648    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-sklearn-s3-tls-serving-fail-predictor-5757db6549-tpx66"]
Oct 30 03:58:18.103423 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103386    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"]
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103705    4377 cpu_manager.go:414] "containerMap: removing container" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103717    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103739    4377 cpu_manager.go:414] "containerMap: removing container" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kube-rbac-proxy"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103745    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kube-rbac-proxy"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103754    4377 cpu_manager.go:414] "containerMap: removing container" podUID="e057af9d-4831-420c-90e4-80732e8690e0" containerName="storage-initializer"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103758    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e057af9d-4831-420c-90e4-80732e8690e0" containerName="storage-initializer"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103768    4377 cpu_manager.go:414] "containerMap: removing container" podUID="e057af9d-4831-420c-90e4-80732e8690e0" containerName="storage-initializer"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103773    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e057af9d-4831-420c-90e4-80732e8690e0" containerName="storage-initializer"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103782    4377 cpu_manager.go:414] "containerMap: removing container" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="storage-initializer"
Oct 30 03:58:18.103826 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103787    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="storage-initializer"
Oct 30 03:58:18.104220 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103833    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kserve-container"
Oct 30 03:58:18.104220 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103844    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea8d8d7-7b62-4e19-8461-336b4802a2fa" containerName="kube-rbac-proxy"
Oct 30 03:58:18.104220 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103853    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="e057af9d-4831-420c-90e4-80732e8690e0" containerName="storage-initializer"
Oct 30 03:58:18.104220 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.103862    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="e057af9d-4831-420c-90e4-80732e8690e0" containerName="storage-initializer"
Oct 30 03:58:18.108363 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.108344    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.124717 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.124695    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:58:18.124827 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.124721    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Oct 30 03:58:18.124902 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.124873    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Oct 30 03:58:18.137675 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.137650    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:58:18.137814 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.137674    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:58:18.137814 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.137700    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:58:18.137814 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.137702    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:58:18.139961 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.139941    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"]
Oct 30 03:58:18.180331 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.180308    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6brg\" (UniqueName: \"kubernetes.io/projected/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-api-access-m6brg\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.180474 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.180362    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.180474 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.180404    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.180589 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.180511    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.180589 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.180574    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392e4ae-cb21-42d8-af03-1647e7a38ff7-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.281492 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.281452    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.281492 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.281496    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.281741 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.281537    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.281741 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.281582    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392e4ae-cb21-42d8-af03-1647e7a38ff7-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.281741 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.281611    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6brg\" (UniqueName: \"kubernetes.io/projected/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-api-access-m6brg\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.282014 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.281987    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.282260 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.282239    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.282317 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.282240    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.284042 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.284021    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392e4ae-cb21-42d8-af03-1647e7a38ff7-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.305321 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.305300    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6brg\" (UniqueName: \"kubernetes.io/projected/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-api-access-m6brg\") pod \"isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") " pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.425111 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.425026    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:18.514478 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.514446    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e057af9d-4831-420c-90e4-80732e8690e0" path="/var/lib/kubelet/pods/e057af9d-4831-420c-90e4-80732e8690e0/volumes"
Oct 30 03:58:18.737908 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:18.737842    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"]
Oct 30 03:58:18.740651 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:58:18.740621    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5392e4ae_cb21_42d8_af03_1647e7a38ff7.slice/crio-848bafa14275cfd01ffbce408363995d2028606cebfbc01e47816476835e9104 WatchSource:0}: Error finding container 848bafa14275cfd01ffbce408363995d2028606cebfbc01e47816476835e9104: Status 404 returned error can't find the container with id 848bafa14275cfd01ffbce408363995d2028606cebfbc01e47816476835e9104
Oct 30 03:58:19.635442 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:19.635400    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerStarted","Data":"5be4bfa2f67441bd75cb06b9727b7c38505bce959f541bc3d707c5242b8fd6e9"}
Oct 30 03:58:19.635866 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:19.635458    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerStarted","Data":"848bafa14275cfd01ffbce408363995d2028606cebfbc01e47816476835e9104"}
Oct 30 03:58:23.652155 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:23.652074    4377 generic.go:356] "Generic (PLEG): container finished" podID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerID="5be4bfa2f67441bd75cb06b9727b7c38505bce959f541bc3d707c5242b8fd6e9" exitCode=0
Oct 30 03:58:23.652557 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:23.652154    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerDied","Data":"5be4bfa2f67441bd75cb06b9727b7c38505bce959f541bc3d707c5242b8fd6e9"}
Oct 30 03:58:23.653473 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:23.653455    4377 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 03:58:35.714020 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:35.713985    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerStarted","Data":"fbd0e7d855902b826b6691b8182b394eb8028b2c740c9c4a0767b3f3594df3be"}
Oct 30 03:58:35.714490 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:35.714046    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerStarted","Data":"96c95f35d27e85b09a76fd8a1604e11405ed20a4335153028a9654b81eb82ae4"}
Oct 30 03:58:35.714490 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:35.714313    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:35.714640 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:35.714483    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:35.715830 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:35.715806    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:58:35.773220 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:35.773170    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podStartSLOduration=1.5899012689999998 podStartE2EDuration="17.773155622s" totalImagesPullingTime="11.589327488s" totalInitContainerRuntime="4.593926865s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:58:35.771735744 +0000 UTC m=+3815.364064550" watchObservedRunningTime="2026-10-30 03:58:35.773155622 +0000 UTC m=+3815.365484427"
Oct 30 03:58:36.718555 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:36.718508    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:58:40.258872 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:58:40.258834    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:58:41.723726 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:41.723695    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:58:41.724270 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:41.724245    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:58:51.725282 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:58:51.725199    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:59:01.724642 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:01.724600    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:59:10.259389 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:59:10.259351    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:59:11.724758 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:11.724722    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:59:21.724474 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:21.724434    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:59:31.725711 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:31.725678    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:59:40.064446 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:40.064413    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"]
Oct 30 03:59:40.064895 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:40.064649    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" containerID="cri-o://96c95f35d27e85b09a76fd8a1604e11405ed20a4335153028a9654b81eb82ae4" gracePeriod=30
Oct 30 03:59:40.064895 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:40.064708    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kube-rbac-proxy" containerID="cri-o://fbd0e7d855902b826b6691b8182b394eb8028b2c740c9c4a0767b3f3594df3be" gracePeriod=30
Oct 30 03:59:40.257814 ip-10-0-138-200 kubenswrapper[4377]: E1030 03:59:40.257771    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 03:59:40.980858 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:40.980821    4377 generic.go:356] "Generic (PLEG): container finished" podID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerID="fbd0e7d855902b826b6691b8182b394eb8028b2c740c9c4a0767b3f3594df3be" exitCode=0
Oct 30 03:59:40.981104 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:40.980881    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerDied","Data":"fbd0e7d855902b826b6691b8182b394eb8028b2c740c9c4a0767b3f3594df3be"}
Oct 30 03:59:41.719675 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:41.719630    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Oct 30 03:59:41.724992 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:41.724957    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Oct 30 03:59:42.996658 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:42.996622    4377 generic.go:356] "Generic (PLEG): container finished" podID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerID="96c95f35d27e85b09a76fd8a1604e11405ed20a4335153028a9654b81eb82ae4" exitCode=0
Oct 30 03:59:42.996998 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:42.996672    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerDied","Data":"96c95f35d27e85b09a76fd8a1604e11405ed20a4335153028a9654b81eb82ae4"}
Oct 30 03:59:42.996998 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:42.996762    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz" event={"ID":"5392e4ae-cb21-42d8-af03-1647e7a38ff7","Type":"ContainerDied","Data":"848bafa14275cfd01ffbce408363995d2028606cebfbc01e47816476835e9104"}
Oct 30 03:59:42.996998 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:42.996785    4377 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="848bafa14275cfd01ffbce408363995d2028606cebfbc01e47816476835e9104"
Oct 30 03:59:42.998885 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:42.998869    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:59:43.103765 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.103692    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5392e4ae-cb21-42d8-af03-1647e7a38ff7-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392e4ae-cb21-42d8-af03-1647e7a38ff7-proxy-tls\") pod \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") "
Oct 30 03:59:43.103765 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.103728    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-api-access-m6brg\" (UniqueName: \"kubernetes.io/projected/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-api-access-m6brg\") pod \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") "
Oct 30 03:59:43.103949 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.103772    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kserve-provision-location\") pod \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") "
Oct 30 03:59:43.103949 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.103808    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-rbac-proxy-sar-config\") pod \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") "
Oct 30 03:59:43.103949 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.103859    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-cabundle-cert\") pod \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\" (UID: \"5392e4ae-cb21-42d8-af03-1647e7a38ff7\") "
Oct 30 03:59:43.104185 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.104153    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-rbac-proxy-sar-config" pod "5392e4ae-cb21-42d8-af03-1647e7a38ff7" (UID: "5392e4ae-cb21-42d8-af03-1647e7a38ff7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:43.104310 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.104223    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-cabundle-cert" pod "5392e4ae-cb21-42d8-af03-1647e7a38ff7" (UID: "5392e4ae-cb21-42d8-af03-1647e7a38ff7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 03:59:43.105877 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.105853    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-api-access-m6brg" pod "5392e4ae-cb21-42d8-af03-1647e7a38ff7" (UID: "5392e4ae-cb21-42d8-af03-1647e7a38ff7"). InnerVolumeSpecName "kube-api-access-m6brg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 03:59:43.105945 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.105875    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5392e4ae-cb21-42d8-af03-1647e7a38ff7-proxy-tls" pod "5392e4ae-cb21-42d8-af03-1647e7a38ff7" (UID: "5392e4ae-cb21-42d8-af03-1647e7a38ff7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 03:59:43.113033 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.113010    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kserve-provision-location" pod "5392e4ae-cb21-42d8-af03-1647e7a38ff7" (UID: "5392e4ae-cb21-42d8-af03-1647e7a38ff7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 03:59:43.205481 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.205457    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5392e4ae-cb21-42d8-af03-1647e7a38ff7-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:59:43.205481 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.205479    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6brg\" (UniqueName: \"kubernetes.io/projected/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-api-access-m6brg\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:59:43.205646 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.205490    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:59:43.205646 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.205499    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:59:43.205646 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:43.205508    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5392e4ae-cb21-42d8-af03-1647e7a38ff7-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 03:59:44.001107 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:44.001075    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"
Oct 30 03:59:44.054451 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:44.054416    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"]
Oct 30 03:59:44.067188 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:44.067155    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-paddle-v2-kserve-predictor-7bc66b67b6-kptxz"]
Oct 30 03:59:44.514327 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:44.514294    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" path="/var/lib/kubelet/pods/5392e4ae-cb21-42d8-af03-1647e7a38ff7/volumes"
Oct 30 03:59:45.584168 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584128    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"]
Oct 30 03:59:45.584712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584595    4377 cpu_manager.go:414] "containerMap: removing container" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container"
Oct 30 03:59:45.584712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584614    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container"
Oct 30 03:59:45.584712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584650    4377 cpu_manager.go:414] "containerMap: removing container" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kube-rbac-proxy"
Oct 30 03:59:45.584712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584658    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kube-rbac-proxy"
Oct 30 03:59:45.584712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584678    4377 cpu_manager.go:414] "containerMap: removing container" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="storage-initializer"
Oct 30 03:59:45.584712 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584686    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="storage-initializer"
Oct 30 03:59:45.585024 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584783    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kube-rbac-proxy"
Oct 30 03:59:45.585024 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.584799    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="5392e4ae-cb21-42d8-af03-1647e7a38ff7" containerName="kserve-container"
Oct 30 03:59:45.589231 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.589210    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.605910 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.605881    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 03:59:45.612873 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.612848    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Oct 30 03:59:45.612873 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.612848    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 03:59:45.613036 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.612904    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 03:59:45.617645 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.617623    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 03:59:45.617770 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.617721    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-pmml-predictor-serving-cert\""
Oct 30 03:59:45.617842 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.617823    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 03:59:45.619448 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.619430    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"]
Oct 30 03:59:45.626500 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.626481    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d57vq\" (UniqueName: \"kubernetes.io/projected/13415397-c95f-486e-8e55-0d5816b753c8-kube-api-access-d57vq\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.626601 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.626510    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-cabundle-cert\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.626641 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.626622    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13415397-c95f-486e-8e55-0d5816b753c8-kserve-provision-location\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.626686 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.626668    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13415397-c95f-486e-8e55-0d5816b753c8-proxy-tls\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.626800 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.626779    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.727432 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.727393    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.727432 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.727438    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d57vq\" (UniqueName: \"kubernetes.io/projected/13415397-c95f-486e-8e55-0d5816b753c8-kube-api-access-d57vq\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.727710 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.727487    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-cabundle-cert\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.727710 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.727574    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13415397-c95f-486e-8e55-0d5816b753c8-kserve-provision-location\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.727710 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.727672    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13415397-c95f-486e-8e55-0d5816b753c8-proxy-tls\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.728061 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.728039    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13415397-c95f-486e-8e55-0d5816b753c8-kserve-provision-location\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.728213 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.728197    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.728291 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.728271    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-cabundle-cert\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.729988 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.729969    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13415397-c95f-486e-8e55-0d5816b753c8-proxy-tls\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.751432 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.751409    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d57vq\" (UniqueName: \"kubernetes.io/projected/13415397-c95f-486e-8e55-0d5816b753c8-kube-api-access-d57vq\") pod \"isvc-pmml-predictor-6ff86c7895-hxz2f\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:45.905948 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:45.905862    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:46.217803 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:46.217776    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"]
Oct 30 03:59:46.220231 ip-10-0-138-200 kubenswrapper[4377]: W1030 03:59:46.220203    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13415397_c95f_486e_8e55_0d5816b753c8.slice/crio-cc4f8ca4d05e7519bb37eb6c2df0c3640e0710d9fe11ffc13c4adb76b79e0b30 WatchSource:0}: Error finding container cc4f8ca4d05e7519bb37eb6c2df0c3640e0710d9fe11ffc13c4adb76b79e0b30: Status 404 returned error can't find the container with id cc4f8ca4d05e7519bb37eb6c2df0c3640e0710d9fe11ffc13c4adb76b79e0b30
Oct 30 03:59:47.015086 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:47.015043    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerStarted","Data":"eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345"}
Oct 30 03:59:47.015477 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:47.015106    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerStarted","Data":"cc4f8ca4d05e7519bb37eb6c2df0c3640e0710d9fe11ffc13c4adb76b79e0b30"}
Oct 30 03:59:51.031697 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:51.031660    4377 generic.go:356] "Generic (PLEG): container finished" podID="13415397-c95f-486e-8e55-0d5816b753c8" containerID="eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345" exitCode=0
Oct 30 03:59:51.032126 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:51.031718    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerDied","Data":"eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345"}
Oct 30 03:59:59.078303 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:59.078264    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerStarted","Data":"0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8"}
Oct 30 03:59:59.078808 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:59.078331    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerStarted","Data":"8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23"}
Oct 30 03:59:59.078808 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:59.078575    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:59.078808 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:59.078630    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 03:59:59.080011 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:59.079981    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 03:59:59.125930 ip-10-0-138-200 kubenswrapper[4377]: I1030 03:59:59.125882    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podStartSLOduration=2.840902829 podStartE2EDuration="14.125868563s" totalImagesPullingTime="7.439767196s" totalInitContainerRuntime="3.845198538s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:59:59.125164575 +0000 UTC m=+3898.717493380" watchObservedRunningTime="2026-10-30 03:59:59.125868563 +0000 UTC m=+3898.718197369"
Oct 30 04:00:00.083432 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:00.083390    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:00:05.087481 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:05.087452    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 04:00:05.087965 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:05.087938    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:00:10.258122 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:00:10.258076    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:00:15.088039 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:15.087951    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:00:25.088701 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:25.088660    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:00:35.088624 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:35.088583    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:00:40.259045 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:00:40.259010    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:00:45.088611 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:45.088575    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:00:55.088082 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:00:55.088040    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:01:05.088762 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:05.088720    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:01:07.509976 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:07.509929    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:01:10.259875 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:01:10.259835    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:01:17.510762 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:17.510717    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Oct 30 04:01:27.511532 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:27.511488    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 04:01:37.809055 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:37.809025    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"]
Oct 30 04:01:37.809645 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:37.809243    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container" containerID="cri-o://8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23" gracePeriod=30
Oct 30 04:01:37.809645 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:37.809283    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kube-rbac-proxy" containerID="cri-o://0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8" gracePeriod=30
Oct 30 04:01:38.477260 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:38.477226    4377 generic.go:356] "Generic (PLEG): container finished" podID="13415397-c95f-486e-8e55-0d5816b753c8" containerID="0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8" exitCode=0
Oct 30 04:01:38.477430 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:38.477282    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerDied","Data":"0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8"}
Oct 30 04:01:40.084479 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:40.084432    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Oct 30 04:01:40.258653 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:01:40.258617    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:01:41.950471 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.950443    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 04:01:41.979456 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979423    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13415397-c95f-486e-8e55-0d5816b753c8-proxy-tls\" (UniqueName: \"kubernetes.io/secret/13415397-c95f-486e-8e55-0d5816b753c8-proxy-tls\") pod \"13415397-c95f-486e-8e55-0d5816b753c8\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") "
Oct 30 04:01:41.979636 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979474    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-cabundle-cert\") pod \"13415397-c95f-486e-8e55-0d5816b753c8\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") "
Oct 30 04:01:41.979636 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979615    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13415397-c95f-486e-8e55-0d5816b753c8-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13415397-c95f-486e-8e55-0d5816b753c8-kserve-provision-location\") pod \"13415397-c95f-486e-8e55-0d5816b753c8\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") "
Oct 30 04:01:41.979761 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979674    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13415397-c95f-486e-8e55-0d5816b753c8-kube-api-access-d57vq\" (UniqueName: \"kubernetes.io/projected/13415397-c95f-486e-8e55-0d5816b753c8-kube-api-access-d57vq\") pod \"13415397-c95f-486e-8e55-0d5816b753c8\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") "
Oct 30 04:01:41.979761 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979714    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-kube-rbac-proxy-sar-config\") pod \"13415397-c95f-486e-8e55-0d5816b753c8\" (UID: \"13415397-c95f-486e-8e55-0d5816b753c8\") "
Oct 30 04:01:41.979871 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979834    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-cabundle-cert" pod "13415397-c95f-486e-8e55-0d5816b753c8" (UID: "13415397-c95f-486e-8e55-0d5816b753c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:41.979912 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979870    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13415397-c95f-486e-8e55-0d5816b753c8-kserve-provision-location" pod "13415397-c95f-486e-8e55-0d5816b753c8" (UID: "13415397-c95f-486e-8e55-0d5816b753c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:01:41.979955 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.979935    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:01:41.980166 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.980141    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-kube-rbac-proxy-sar-config" pod "13415397-c95f-486e-8e55-0d5816b753c8" (UID: "13415397-c95f-486e-8e55-0d5816b753c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:01:41.981631 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.981605    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13415397-c95f-486e-8e55-0d5816b753c8-proxy-tls" pod "13415397-c95f-486e-8e55-0d5816b753c8" (UID: "13415397-c95f-486e-8e55-0d5816b753c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:01:41.981756 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:41.981652    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13415397-c95f-486e-8e55-0d5816b753c8-kube-api-access-d57vq" pod "13415397-c95f-486e-8e55-0d5816b753c8" (UID: "13415397-c95f-486e-8e55-0d5816b753c8"). InnerVolumeSpecName "kube-api-access-d57vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:01:42.081233 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.081152    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13415397-c95f-486e-8e55-0d5816b753c8-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:01:42.081233 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.081180    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13415397-c95f-486e-8e55-0d5816b753c8-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:01:42.081233 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.081189    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d57vq\" (UniqueName: \"kubernetes.io/projected/13415397-c95f-486e-8e55-0d5816b753c8-kube-api-access-d57vq\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:01:42.081233 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.081199    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13415397-c95f-486e-8e55-0d5816b753c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:01:42.497130 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.497042    4377 generic.go:356] "Generic (PLEG): container finished" podID="13415397-c95f-486e-8e55-0d5816b753c8" containerID="8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23" exitCode=0
Oct 30 04:01:42.497130 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.497091    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerDied","Data":"8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23"}
Oct 30 04:01:42.497130 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.497101    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"
Oct 30 04:01:42.497361 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.497168    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f" event={"ID":"13415397-c95f-486e-8e55-0d5816b753c8","Type":"ContainerDied","Data":"cc4f8ca4d05e7519bb37eb6c2df0c3640e0710d9fe11ffc13c4adb76b79e0b30"}
Oct 30 04:01:42.497361 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.497194    4377 scope.go:122] "RemoveContainer" containerID="0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8"
Oct 30 04:01:42.505541 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.505506    4377 scope.go:122] "RemoveContainer" containerID="8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23"
Oct 30 04:01:42.512961 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.512942    4377 scope.go:122] "RemoveContainer" containerID="eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345"
Oct 30 04:01:42.520162 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.520126    4377 scope.go:122] "RemoveContainer" containerID="0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8"
Oct 30 04:01:42.520413 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:01:42.520393    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8\": container with ID starting with 0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8 not found: ID does not exist" containerID="0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8"
Oct 30 04:01:42.520489 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.520429    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8"} err="failed to get container status \"0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8\": rpc error: code = NotFound desc = could not find container \"0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8\": container with ID starting with 0d0dbc8fbc3a890f7fb9dc388666ce8bbcd0a6ea3b552f307e3f5789c2a878e8 not found: ID does not exist"
Oct 30 04:01:42.520489 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.520453    4377 scope.go:122] "RemoveContainer" containerID="8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23"
Oct 30 04:01:42.520747 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:01:42.520725    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23\": container with ID starting with 8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23 not found: ID does not exist" containerID="8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23"
Oct 30 04:01:42.520855 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.520755    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23"} err="failed to get container status \"8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23\": rpc error: code = NotFound desc = could not find container \"8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23\": container with ID starting with 8d0c5f311dd4eee4082f3a33e2b6ce409989c187a6ba7859fd456ca8e5763e23 not found: ID does not exist"
Oct 30 04:01:42.520855 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.520778    4377 scope.go:122] "RemoveContainer" containerID="eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345"
Oct 30 04:01:42.521008 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:01:42.520988    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345\": container with ID starting with eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345 not found: ID does not exist" containerID="eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345"
Oct 30 04:01:42.521067 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.521018    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345"} err="failed to get container status \"eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345\": rpc error: code = NotFound desc = could not find container \"eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345\": container with ID starting with eebd71319763cf9201bbed93bec5e18f9dcc23e90564971b4a65127252f95345 not found: ID does not exist"
Oct 30 04:01:42.551306 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.551275    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"]
Oct 30 04:01:42.563952 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:42.563927    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-predictor-6ff86c7895-hxz2f"]
Oct 30 04:01:43.350671 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.350642    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"]
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.350958    4377 cpu_manager.go:414] "containerMap: removing container" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="storage-initializer"
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.350970    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="storage-initializer"
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.350985    4377 cpu_manager.go:414] "containerMap: removing container" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kube-rbac-proxy"
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.350992    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kube-rbac-proxy"
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.351001    4377 cpu_manager.go:414] "containerMap: removing container" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container"
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.351006    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container"
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.351069    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kube-rbac-proxy"
Oct 30 04:01:43.351163 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.351077    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="13415397-c95f-486e-8e55-0d5816b753c8" containerName="kserve-container"
Oct 30 04:01:43.355619 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.355601    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.371879 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.371854    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"seaweedfs-s3-creds\""
Oct 30 04:01:43.384775 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.384619    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"kube-root-ca.crt\""
Oct 30 04:01:43.384775 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.384688    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"openshift-service-ca.crt\""
Oct 30 04:01:43.384775 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.384693    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Oct 30 04:01:43.384775 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.384708    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"default-dockercfg-5tpp2\""
Oct 30 04:01:43.384775 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.384695    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-773c71d3d4\"/\"odh-kserve-custom-ca-bundle\""
Oct 30 04:01:43.385153 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.384816    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-773c71d3d4\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Oct 30 04:01:43.386057 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.386036    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"]
Oct 30 04:01:43.393193 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.393172    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e4eda2-0be0-4c16-88db-cac561bade52-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.393276 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.393202    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.393276 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.393247    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.393355 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.393341    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e4eda2-0be0-4c16-88db-cac561bade52-proxy-tls\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.393388 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.393367    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxh9n\" (UniqueName: \"kubernetes.io/projected/17e4eda2-0be0-4c16-88db-cac561bade52-kube-api-access-jxh9n\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.494482 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.494449    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e4eda2-0be0-4c16-88db-cac561bade52-proxy-tls\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.494651 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.494486    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxh9n\" (UniqueName: \"kubernetes.io/projected/17e4eda2-0be0-4c16-88db-cac561bade52-kube-api-access-jxh9n\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.494722 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.494693    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e4eda2-0be0-4c16-88db-cac561bade52-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.494761 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.494736    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.494810 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.494794    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.495152 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.495121    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e4eda2-0be0-4c16-88db-cac561bade52-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.495432 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.495416    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.495432 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.495427    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.496961 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.496940    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e4eda2-0be0-4c16-88db-cac561bade52-proxy-tls\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.518239 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.518214    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxh9n\" (UniqueName: \"kubernetes.io/projected/17e4eda2-0be0-4c16-88db-cac561bade52-kube-api-access-jxh9n\") pod \"isvc-pmml-runtime-predictor-84bd9b7f69-29lxp\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") " pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.672164 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.672073    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:43.983261 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:43.983234    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"]
Oct 30 04:01:43.985879 ip-10-0-138-200 kubenswrapper[4377]: W1030 04:01:43.985853    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e4eda2_0be0_4c16_88db_cac561bade52.slice/crio-185550019be1830a46b3bfcc65acc0402bbb8825e0421f58fd6565a246ef478e WatchSource:0}: Error finding container 185550019be1830a46b3bfcc65acc0402bbb8825e0421f58fd6565a246ef478e: Status 404 returned error can't find the container with id 185550019be1830a46b3bfcc65acc0402bbb8825e0421f58fd6565a246ef478e
Oct 30 04:01:44.513234 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:44.513201    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13415397-c95f-486e-8e55-0d5816b753c8" path="/var/lib/kubelet/pods/13415397-c95f-486e-8e55-0d5816b753c8/volumes"
Oct 30 04:01:44.513613 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:44.513544    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerStarted","Data":"07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3"}
Oct 30 04:01:44.513613 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:44.513598    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerStarted","Data":"185550019be1830a46b3bfcc65acc0402bbb8825e0421f58fd6565a246ef478e"}
Oct 30 04:01:48.526942 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:48.526904    4377 generic.go:356] "Generic (PLEG): container finished" podID="17e4eda2-0be0-4c16-88db-cac561bade52" containerID="07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3" exitCode=0
Oct 30 04:01:48.527314 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:48.526977    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerDied","Data":"07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3"}
Oct 30 04:01:49.533564 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:49.533509    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerStarted","Data":"bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38"}
Oct 30 04:01:49.534055 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:49.533626    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerStarted","Data":"0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe"}
Oct 30 04:01:49.534055 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:49.533852    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:49.592324 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:49.592278    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podStartSLOduration=2.892464608 podStartE2EDuration="6.592266371s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699801763s" isStatefulPod=false podCreationTimestamp="2026-10-30 04:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:01:49.591881361 +0000 UTC m=+4009.184210168" watchObservedRunningTime="2026-10-30 04:01:49.592266371 +0000 UTC m=+4009.184595177"
Oct 30 04:01:50.538067 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:50.538034    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:50.539467 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:50.539441    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:01:51.542879 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:51.542841    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:01:56.547569 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:56.547536    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:01:56.548051 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:01:56.548016    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:02:06.548139 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:02:06.548097    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:02:10.258776 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:02:10.258738    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:02:16.548895 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:02:16.548854    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:02:26.548151 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:02:26.548108    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:02:36.548187 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:02:36.548151    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:02:40.259130 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:02:40.259095    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:02:46.548848 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:02:46.548808    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:02:56.548839 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:02:56.548799    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:03:06.547977 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:06.547928    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:03:10.259036 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:03:10.258998    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:03:16.548620 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:16.548541    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:03:25.210053 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:25.210018    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"]
Oct 30 04:03:25.210597 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:25.210322    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" containerID="cri-o://0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe" gracePeriod=30
Oct 30 04:03:25.210597 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:25.210348    4377 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kube-rbac-proxy" containerID="cri-o://bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38" gracePeriod=30
Oct 30 04:03:25.924615 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:25.924583    4377 generic.go:356] "Generic (PLEG): container finished" podID="17e4eda2-0be0-4c16-88db-cac561bade52" containerID="bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38" exitCode=0
Oct 30 04:03:25.924806 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:25.924664    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerDied","Data":"bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38"}
Oct 30 04:03:26.543373 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:26.543335    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Oct 30 04:03:26.548715 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:26.548683    4377 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Oct 30 04:03:29.447072 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.447043    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:03:29.467872 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.467846    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e4eda2-0be0-4c16-88db-cac561bade52-proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e4eda2-0be0-4c16-88db-cac561bade52-proxy-tls\") pod \"17e4eda2-0be0-4c16-88db-cac561bade52\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") "
Oct 30 04:03:29.468038 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.467893    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e4eda2-0be0-4c16-88db-cac561bade52-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e4eda2-0be0-4c16-88db-cac561bade52-kserve-provision-location\") pod \"17e4eda2-0be0-4c16-88db-cac561bade52\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") "
Oct 30 04:03:29.468038 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.467922    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-cabundle-cert\") pod \"17e4eda2-0be0-4c16-88db-cac561bade52\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") "
Oct 30 04:03:29.468038 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.467956    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e4eda2-0be0-4c16-88db-cac561bade52-kube-api-access-jxh9n\" (UniqueName: \"kubernetes.io/projected/17e4eda2-0be0-4c16-88db-cac561bade52-kube-api-access-jxh9n\") pod \"17e4eda2-0be0-4c16-88db-cac561bade52\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") "
Oct 30 04:03:29.468290 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.468258    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e4eda2-0be0-4c16-88db-cac561bade52-kserve-provision-location" pod "17e4eda2-0be0-4c16-88db-cac561bade52" (UID: "17e4eda2-0be0-4c16-88db-cac561bade52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 04:03:29.468436 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.468296    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-cabundle-cert" pod "17e4eda2-0be0-4c16-88db-cac561bade52" (UID: "17e4eda2-0be0-4c16-88db-cac561bade52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:29.469995 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.469965    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e4eda2-0be0-4c16-88db-cac561bade52-kube-api-access-jxh9n" pod "17e4eda2-0be0-4c16-88db-cac561bade52" (UID: "17e4eda2-0be0-4c16-88db-cac561bade52"). InnerVolumeSpecName "kube-api-access-jxh9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 04:03:29.470081 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.469999    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e4eda2-0be0-4c16-88db-cac561bade52-proxy-tls" pod "17e4eda2-0be0-4c16-88db-cac561bade52" (UID: "17e4eda2-0be0-4c16-88db-cac561bade52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 04:03:29.568758 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.568676    4377 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-kube-rbac-proxy-sar-config\") pod \"17e4eda2-0be0-4c16-88db-cac561bade52\" (UID: \"17e4eda2-0be0-4c16-88db-cac561bade52\") "
Oct 30 04:03:29.568907 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.568867    4377 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/17e4eda2-0be0-4c16-88db-cac561bade52-proxy-tls\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:03:29.568907 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.568880    4377 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/17e4eda2-0be0-4c16-88db-cac561bade52-kserve-provision-location\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:03:29.568907 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.568890    4377 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-cabundle-cert\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:03:29.568907 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.568899    4377 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxh9n\" (UniqueName: \"kubernetes.io/projected/17e4eda2-0be0-4c16-88db-cac561bade52-kube-api-access-jxh9n\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:03:29.569045 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.568970    4377 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-kube-rbac-proxy-sar-config" pod "17e4eda2-0be0-4c16-88db-cac561bade52" (UID: "17e4eda2-0be0-4c16-88db-cac561bade52"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 04:03:29.669977 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.669944    4377 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/17e4eda2-0be0-4c16-88db-cac561bade52-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-200.ec2.internal\" DevicePath \"\""
Oct 30 04:03:29.945680 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.945592    4377 generic.go:356] "Generic (PLEG): container finished" podID="17e4eda2-0be0-4c16-88db-cac561bade52" containerID="0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe" exitCode=0
Oct 30 04:03:29.945680 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.945652    4377 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"
Oct 30 04:03:29.945914 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.945679    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerDied","Data":"0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe"}
Oct 30 04:03:29.945914 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.945753    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp" event={"ID":"17e4eda2-0be0-4c16-88db-cac561bade52","Type":"ContainerDied","Data":"185550019be1830a46b3bfcc65acc0402bbb8825e0421f58fd6565a246ef478e"}
Oct 30 04:03:29.945914 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.945790    4377 scope.go:122] "RemoveContainer" containerID="bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38"
Oct 30 04:03:29.954598 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.954575    4377 scope.go:122] "RemoveContainer" containerID="0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe"
Oct 30 04:03:29.962133 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.962115    4377 scope.go:122] "RemoveContainer" containerID="07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3"
Oct 30 04:03:29.969011 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.968994    4377 scope.go:122] "RemoveContainer" containerID="bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38"
Oct 30 04:03:29.969296 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:03:29.969275    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38\": container with ID starting with bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38 not found: ID does not exist" containerID="bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38"
Oct 30 04:03:29.969364 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.969304    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38"} err="failed to get container status \"bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38\": rpc error: code = NotFound desc = could not find container \"bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38\": container with ID starting with bc277f2ebdb55e4dfa7627905ea148bef2cb08c6112911e7f9c5da27866dcf38 not found: ID does not exist"
Oct 30 04:03:29.969364 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.969321    4377 scope.go:122] "RemoveContainer" containerID="0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe"
Oct 30 04:03:29.969699 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:03:29.969611    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe\": container with ID starting with 0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe not found: ID does not exist" containerID="0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe"
Oct 30 04:03:29.969699 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.969635    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe"} err="failed to get container status \"0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe\": rpc error: code = NotFound desc = could not find container \"0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe\": container with ID starting with 0685cb31ed1b74c8b406cf508247c9d226599e7693f4b51e24052a93b84033fe not found: ID does not exist"
Oct 30 04:03:29.969699 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.969653    4377 scope.go:122] "RemoveContainer" containerID="07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3"
Oct 30 04:03:29.969925 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:03:29.969897    4377 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3\": container with ID starting with 07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3 not found: ID does not exist" containerID="07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3"
Oct 30 04:03:29.969968 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.969933    4377 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3"} err="failed to get container status \"07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3\": rpc error: code = NotFound desc = could not find container \"07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3\": container with ID starting with 07828fb7c0c028d081c49cb5b2ab7ee3f7f21af3c21400bddef375332b4e10c3 not found: ID does not exist"
Oct 30 04:03:29.999016 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:29.998995    4377 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"]
Oct 30 04:03:30.011452 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:30.011433    4377 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-773c71d3d4/isvc-pmml-runtime-predictor-84bd9b7f69-29lxp"]
Oct 30 04:03:30.514268 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:03:30.514237    4377 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" path="/var/lib/kubelet/pods/17e4eda2-0be0-4c16-88db-cac561bade52/volumes"
Oct 30 04:03:40.258411 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:03:40.258371    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:04:10.257978 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:04:10.257937    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:04:16.293241 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:16.293210    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lhtq6_ccbc2f47-f780-45bf-8d91-e2d081bab080/global-pull-secret-syncer/0.log"
Oct 30 04:04:16.975122 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:16.975090    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-j9s8d_3ef68d50-d2d2-4244-848f-208cf21b039a/konnectivity-agent/0.log"
Oct 30 04:04:17.228604 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:17.228495    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-200.ec2.internal_fd03e27bf6704eb151f6eb79c368f447/haproxy/0.log"
Oct 30 04:04:23.353685 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:23.353656    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z5qvn_9dbb7bb1-9c6c-4eef-824a-84a3adccbe35/kube-state-metrics/0.log"
Oct 30 04:04:23.470069 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:23.470033    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z5qvn_9dbb7bb1-9c6c-4eef-824a-84a3adccbe35/kube-rbac-proxy-main/0.log"
Oct 30 04:04:23.581805 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:23.581782    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-z5qvn_9dbb7bb1-9c6c-4eef-824a-84a3adccbe35/kube-rbac-proxy-self/0.log"
Oct 30 04:04:23.709235 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:23.709164    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-677996c9cc-w657x_f05cafde-12ba-4845-9ad7-8f91be4e3ba8/metrics-server/0.log"
Oct 30 04:04:23.941080 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:23.941052    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wnpc_eafa9117-aa83-435f-bdc4-9652062d448e/node-exporter/0.log"
Oct 30 04:04:24.054408 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:24.054379    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wnpc_eafa9117-aa83-435f-bdc4-9652062d448e/kube-rbac-proxy/0.log"
Oct 30 04:04:24.170487 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:24.170463    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5wnpc_eafa9117-aa83-435f-bdc4-9652062d448e/init-textfile/0.log"
Oct 30 04:04:25.418180 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418140    4377 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"]
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418608    4377 cpu_manager.go:414] "containerMap: removing container" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container"
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418626    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container"
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418649    4377 cpu_manager.go:414] "containerMap: removing container" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kube-rbac-proxy"
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418656    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kube-rbac-proxy"
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418686    4377 cpu_manager.go:414] "containerMap: removing container" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="storage-initializer"
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418694    4377 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="storage-initializer"
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418766    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kserve-container"
Oct 30 04:04:25.418818 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.418779    4377 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e4eda2-0be0-4c16-88db-cac561bade52" containerName="kube-rbac-proxy"
Oct 30 04:04:25.421839 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.421818    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.424218 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.424186    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svsqx\" (UniqueName: \"kubernetes.io/projected/8a55ee62-3693-460c-880b-570bd7f90507-kube-api-access-svsqx\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.424328 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.424235    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-lib-modules\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.424380 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.424334    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-sys\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.424437 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.424422    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-proc\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.424477 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.424446    4377 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-podres\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.440793 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.440764    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"]
Oct 30 04:04:25.452045 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.452018    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4bss\"/\"openshift-service-ca.crt\""
Oct 30 04:04:25.452045 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.452041    4377 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s4bss\"/\"default-dockercfg-gs6td\""
Oct 30 04:04:25.452643 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.452113    4377 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4bss\"/\"kube-root-ca.crt\""
Oct 30 04:04:25.525202 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525163    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-proc\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525202 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525204    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-podres\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525461 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525231    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svsqx\" (UniqueName: \"kubernetes.io/projected/8a55ee62-3693-460c-880b-570bd7f90507-kube-api-access-svsqx\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525461 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525259    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-lib-modules\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525461 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525325    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-podres\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525461 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525340    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-proc\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525461 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525338    4377 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-sys\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525461 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525388    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-sys\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.525461 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.525448    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a55ee62-3693-460c-880b-570bd7f90507-lib-modules\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.549458 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.549431    4377 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svsqx\" (UniqueName: \"kubernetes.io/projected/8a55ee62-3693-460c-880b-570bd7f90507-kube-api-access-svsqx\") pod \"perf-node-gather-daemonset-s94w7\" (UID: \"8a55ee62-3693-460c-880b-570bd7f90507\") " pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:25.734709 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:25.734611    4377 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:26.030724 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.030562    4377 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"]
Oct 30 04:04:26.033540 ip-10-0-138-200 kubenswrapper[4377]: W1030 04:04:26.033501    4377 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a55ee62_3693_460c_880b_570bd7f90507.slice/crio-9a50ed04ae2f153459f38452c508cd8fbe07948cf4855c7c35b9bf652070b071 WatchSource:0}: Error finding container 9a50ed04ae2f153459f38452c508cd8fbe07948cf4855c7c35b9bf652070b071: Status 404 returned error can't find the container with id 9a50ed04ae2f153459f38452c508cd8fbe07948cf4855c7c35b9bf652070b071
Oct 30 04:04:26.169761 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.169717    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7" event={"ID":"8a55ee62-3693-460c-880b-570bd7f90507","Type":"ContainerStarted","Data":"b9712a9f86a90351c480f8c47a75b7d09d31d407ac510170d15bf821a2aebcaa"}
Oct 30 04:04:26.169954 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.169779    4377 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7" event={"ID":"8a55ee62-3693-460c-880b-570bd7f90507","Type":"ContainerStarted","Data":"9a50ed04ae2f153459f38452c508cd8fbe07948cf4855c7c35b9bf652070b071"}
Oct 30 04:04:26.169954 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.169815    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:26.211967 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.211915    4377 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7" podStartSLOduration=1.211901375 podStartE2EDuration="1.211901375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 04:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 04:04:26.211378867 +0000 UTC m=+4165.803707689" watchObservedRunningTime="2026-10-30 04:04:26.211901375 +0000 UTC m=+4165.804230180"
Oct 30 04:04:26.382095 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.382013    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2s9lx_1b8014f7-f7ae-4d86-ab01-5d2878c532f7/prometheus-operator-admission-webhook/0.log"
Oct 30 04:04:26.502622 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.502594    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-599ff67b9b-k8vwn_3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4/telemeter-client/0.log"
Oct 30 04:04:26.615765 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.615740    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-599ff67b9b-k8vwn_3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4/reload/0.log"
Oct 30 04:04:26.728603 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:26.728509    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-599ff67b9b-k8vwn_3b6c971f-7f6a-4ed4-80aa-43d7ccce18b4/kube-rbac-proxy/0.log"
Oct 30 04:04:30.481056 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:30.481023    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-795798578b-gxdww_392857bc-41ef-408e-9360-6a5145dd6e5a/console/0.log"
Oct 30 04:04:30.609555 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:30.609510    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-5gzvn_83bf340e-9d8a-4ec7-9308-4a449e878817/download-server/0.log"
Oct 30 04:04:32.183654 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:32.183630    4377 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-s4bss/perf-node-gather-daemonset-s94w7"
Oct 30 04:04:33.642256 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:33.642229    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h9c28_e2252025-152e-4341-9a75-285b38cc2436/dns/0.log"
Oct 30 04:04:33.753082 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:33.753054    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-h9c28_e2252025-152e-4341-9a75-285b38cc2436/kube-rbac-proxy/0.log"
Oct 30 04:04:34.331769 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:34.331735    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-x8h8g_0dd0bf35-fef8-4975-90f7-7bf6c1b70d7f/dns-node-resolver/0.log"
Oct 30 04:04:35.366582 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:35.366550    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56b49fb55b-sgq6q_b593a669-e267-4333-b0e5-72f4a863aa56/registry/0.log"
Oct 30 04:04:35.750919 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:35.750844    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zq2m7_44b96a13-ca71-48c9-9fa1-6d785e9e3b3c/node-ca/0.log"
Oct 30 04:04:38.619599 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:38.619571    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rq99w_6c4663c2-fffd-489d-8f4b-aa26181bff45/serve-healthcheck-canary/0.log"
Oct 30 04:04:40.258977 ip-10-0-138-200 kubenswrapper[4377]: E1030 04:04:40.258939    4377 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jtjnj" containerName="extractor"
Oct 30 04:04:40.270822 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:40.270800    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jtjnj_027c260d-eae4-4dda-834e-be9433d32534/kube-rbac-proxy/0.log"
Oct 30 04:04:40.382663 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:40.382637    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jtjnj_027c260d-eae4-4dda-834e-be9433d32534/exporter/0.log"
Oct 30 04:04:40.494794 ip-10-0-138-200 kubenswrapper[4377]: I1030 04:04:40.494764    4377 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jtjnj_027c260d-eae4-4dda-834e-be9433d32534/extractor/0.log"